Sep 16 13:06:34.436549 ip-10-0-133-73 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 13:06:34.436561 ip-10-0-133-73 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 13:06:34.436568 ip-10-0-133-73 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 13:06:34.436855 ip-10-0-133-73 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 13:06:44.559340 ip-10-0-133-73 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 13:06:44.559358 ip-10-0-133-73 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 72c5fd241fe84a738b415591877dd033 -- Sep 16 13:08:33.627388 ip-10-0-133-73 systemd[1]: Starting Kubernetes Kubelet... Sep 16 13:08:34.059732 ip-10-0-133-73 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:08:34.059732 ip-10-0-133-73 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 13:08:34.059732 ip-10-0-133-73 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:08:34.059732 ip-10-0-133-73 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 13:08:34.059732 ip-10-0-133-73 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:08:34.060373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.059813 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 13:08:34.063826 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063810 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:08:34.063826 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063825 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063828 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063832 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063835 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063837 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063841 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063843 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063846 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063849 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063852 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063854 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063857 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063860 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063863 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063866 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063868 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063871 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063874 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063877 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:08:34.063892 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063880 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063882 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063886 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063890 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063894 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063898 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063901 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063903 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063906 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063909 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063912 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063914 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063917 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063919 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063922 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063924 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063927 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063931 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063934 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:08:34.064356 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063936 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063939 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063941 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063944 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063948 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063951 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063954 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063957 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063961 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063964 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063966 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063969 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063971 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063974 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063977 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063980 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063983 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063986 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063990 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:08:34.064853 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063993 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063995 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.063998 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064001 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064003 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064006 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064008 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064011 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064013 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064016 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064018 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064021 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064024 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064027 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064030 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064032 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064035 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064037 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064039 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064042 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:08:34.065328 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064045 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064047 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064050 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064052 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064055 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064058 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064060 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064063 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064067 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064069 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064072 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064075 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064077 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064080 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064082 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064085 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064087 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064090 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064092 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064095 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:08:34.065809 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064097 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064801 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064807 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064810 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064813 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064816 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064819 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064822 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064825 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064828 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064831 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064833 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064836 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064839 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064841 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064844 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064847 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064851 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064855 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:08:34.066330 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064859 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064863 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064867 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064870 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064873 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064876 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064878 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064881 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064883 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064886 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064889 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064891 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064894 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064897 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064900 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064902 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064905 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064908 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064910 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064913 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:08:34.066815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064916 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064920 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064923 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064926 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064928 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064931 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064934 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064936 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064939 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064941 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064944 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064947 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064949 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064952 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064956 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064958 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064961 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064963 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064966 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:08:34.067313 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064968 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064971 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064973 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064976 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064978 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064981 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064983 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064986 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064988 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064991 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064993 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064996 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.064998 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065001 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065005 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065008 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065011 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065013 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065016 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:08:34.067815 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065019 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065023 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065026 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065029 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065031 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065034 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065037 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065040 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065042 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065045 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065048 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065050 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065053 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065056 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065060 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065063 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065066 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065070 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065073 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065075 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:08:34.068304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065079 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065081 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065084 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065456 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065461 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065464 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065467 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065471 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065473 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065476 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065479 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065481 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065484 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065487 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065489 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065492 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065494 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065497 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065500 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065505 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:08:34.068805 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065507 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065510 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065513 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065516 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065518 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065521 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065524 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065526 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065531 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065535 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065537 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065540 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065543 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065546 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065548 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065551 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065554 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065556 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065559 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:08:34.069304 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065561 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065565 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065567 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065570 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065572 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065575 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065578 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065580 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065582 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065585 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065588 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065590 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065595 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065597 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065600 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065603 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065605 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065610 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065613 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:08:34.069777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065616 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065619 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065622 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065624 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065627 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065630 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065632 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065635 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065638 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065640 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065643 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065646 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065648 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065651 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065655 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065659 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065662 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065665 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065667 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:08:34.070315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065670 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065672 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065675 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065677 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065680 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065682 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065687 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065689 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065692 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065695 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065697 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065700 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065703 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065705 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065708 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065710 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065713 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065715 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065718 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065721 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:08:34.070779 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065723 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065726 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065728 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065731 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.065733 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065810 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065851 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065858 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065863 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065868 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065871 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065875 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065879 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065882 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065885 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065889 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065892 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065895 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065902 2581 flags.go:64] FLAG: --cgroup-root="" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065906 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065909 2581 flags.go:64] FLAG: --client-ca-file="" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065912 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065915 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065919 2581 flags.go:64] FLAG: --cluster-domain="" Sep 16 13:08:34.071300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065922 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065925 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065928 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065931 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065935 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065938 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065942 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065945 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065948 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065951 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065954 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065957 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065960 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065964 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065966 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065969 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065972 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065976 2581 flags.go:64] FLAG: --enable-server="true" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065979 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065984 2581 flags.go:64] FLAG: --event-burst="100" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065987 2581 flags.go:64] FLAG: --event-qps="50" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065990 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065993 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065996 2581 flags.go:64] FLAG: --eviction-hard="" Sep 16 13:08:34.071879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.065999 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066002 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066007 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066010 2581 flags.go:64] FLAG: --eviction-soft="" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066013 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066016 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066019 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066022 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066025 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066028 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066031 2581 flags.go:64] FLAG: --feature-gates="" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066035 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066038 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066041 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066044 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066048 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066051 2581 flags.go:64] FLAG: --help="false" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066054 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066057 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066060 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066063 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066066 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066069 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066072 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 13:08:34.072465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066077 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066080 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066083 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066086 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066089 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066092 2581 flags.go:64] FLAG: --kube-reserved="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066095 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066098 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066101 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066104 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066108 2581 flags.go:64] FLAG: --lock-file="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066111 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066114 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066117 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066135 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066138 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066140 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066143 2581 flags.go:64] FLAG: --logging-format="text" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066147 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066150 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066153 2581 flags.go:64] FLAG: --manifest-url="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066156 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066160 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066162 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066167 2581 flags.go:64] FLAG: --max-pods="110" Sep 16 13:08:34.073094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066170 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066173 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066176 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066179 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066182 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066184 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066187 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066199 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066202 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066205 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066208 2581 flags.go:64] FLAG: --pod-cidr="" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066211 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066216 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066219 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066222 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066225 2581 flags.go:64] FLAG: --port="10250" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066228 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066231 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04bf028a20b81ea1a" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066236 2581 flags.go:64] FLAG: --qos-reserved="" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066239 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066242 2581 flags.go:64] FLAG: --register-node="true" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066245 2581 flags.go:64] FLAG: --register-with-taints="" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066249 2581 flags.go:64] FLAG: --registry-burst="10" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066251 2581 flags.go:64] FLAG: --registry-qps="5" Sep 16 13:08:34.073760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066254 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066257 2581 flags.go:64] FLAG: --reserved-memory="" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066261 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066264 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066266 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066269 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066272 2581 flags.go:64] FLAG: --runonce="false" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066275 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066278 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066281 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066284 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066287 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066290 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066293 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066296 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066300 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066303 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066306 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066308 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066312 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066315 2581 flags.go:64] FLAG: --system-cgroups="" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066318 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066324 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066327 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066330 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 13:08:34.074382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066334 2581 flags.go:64] FLAG: --tls-min-version="" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066338 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066341 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066344 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066347 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066350 2581 flags.go:64] FLAG: --v="2" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066353 2581 flags.go:64] FLAG: --version="false" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066357 2581 flags.go:64] FLAG: --vmodule="" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066361 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066364 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066468 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066472 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066475 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066478 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066480 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066485 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066488 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066491 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066494 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066497 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066500 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:08:34.074981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066503 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066508 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066510 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066513 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066516 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066518 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066521 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066525 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066528 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066530 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066533 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066536 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066539 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066542 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066545 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066547 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066550 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066552 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066555 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066557 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:08:34.075500 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066561 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066563 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066566 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066568 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066571 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066574 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066577 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066581 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066584 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066587 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066590 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066593 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066595 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066599 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066602 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066605 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066608 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066611 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066613 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:08:34.076002 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066616 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066619 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066622 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066625 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066627 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066630 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066633 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066636 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066638 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066641 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066643 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066646 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066649 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066651 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066654 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066657 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066659 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066662 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066664 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066667 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:08:34.076486 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066670 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066672 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066674 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066677 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066680 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066684 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066689 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066691 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066694 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066697 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066699 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066702 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066704 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066715 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066718 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066720 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066724 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066727 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066729 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066732 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:08:34.077016 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066735 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066737 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066740 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066742 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066745 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066747 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066750 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066753 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.066755 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.066760 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.074911 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.074937 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.075955 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.075963 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076024 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:08:34.077509 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076029 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076032 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076036 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076040 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076043 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076046 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076049 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076052 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076055 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076058 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076061 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076063 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076066 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076069 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076071 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076074 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076077 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076079 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076082 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076084 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:08:34.077887 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076087 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076090 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076092 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076095 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076097 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076100 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076102 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076105 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076114 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076117 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076120 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076139 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076142 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076145 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076147 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076150 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076153 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076155 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076158 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076160 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:08:34.078397 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076163 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076166 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076169 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076171 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076174 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076176 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076179 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076182 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076184 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076188 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076192 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076195 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076198 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076200 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076203 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076205 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076208 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076210 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076213 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:08:34.078882 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076215 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076218 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076226 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076229 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076231 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076234 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076237 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076239 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076242 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076244 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076247 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076249 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076252 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076255 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076257 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076259 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076262 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076264 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076268 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076272 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:08:34.079360 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076275 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076278 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076281 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076285 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076288 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076290 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076293 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076296 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076298 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076301 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076303 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076306 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076308 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076311 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076313 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076321 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076324 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076327 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:08:34.079900 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076329 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.076334 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076491 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076496 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076499 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076502 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076505 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076508 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076510 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076512 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076515 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076517 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076520 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076522 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076525 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:08:34.080384 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076528 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076530 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076533 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076536 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076538 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076540 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076543 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076545 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076548 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076550 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076553 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076555 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076558 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076561 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076563 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076571 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076574 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076577 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076579 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076582 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:08:34.080761 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076584 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076587 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076589 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076592 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076594 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076597 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076599 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076602 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076604 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076607 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076609 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076612 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076615 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076617 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076620 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076622 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076625 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076627 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076630 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:08:34.081255 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076634 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076638 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076641 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076643 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076646 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076649 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076652 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076655 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076657 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076665 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076668 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076671 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076673 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076676 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076679 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076681 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076683 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076686 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076689 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076691 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:08:34.081701 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076693 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076696 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076699 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076701 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076703 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076707 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076711 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076714 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076716 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076719 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076721 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076724 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076726 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076729 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076733 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076737 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076741 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076743 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076746 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:08:34.082203 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076748 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076751 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076754 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076761 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076764 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076766 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076769 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:34.076771 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.076776 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.076898 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.079397 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.080472 2581 server.go:1019] "Starting client certificate rotation" Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.080575 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:08:34.082661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.080613 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:08:34.083293 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.083279 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:08:34.087314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.087256 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:08:34.094333 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.094316 2581 log.go:25] "Validated CRI v1 runtime API" Sep 16 13:08:34.097768 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.097752 2581 log.go:25] "Validated CRI v1 image API" Sep 16 13:08:34.098559 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.098544 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 13:08:34.103504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.103487 2581 fs.go:135] Filesystem UUIDs: map[067841ba-8e02-490b-bf9f-aa64c72d53d6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f270d95d-151e-49fc-b3fb-389132f44dac:/dev/nvme0n1p4] Sep 16 13:08:34.103550 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.103504 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 13:08:34.108611 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.108513 2581 manager.go:216] Machine: {Timestamp:2026-09-16 13:08:34.106894972 +0000 UTC m=+0.371779193 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119295 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263346a990496e56d5210dcfc00880 SystemUUID:ec263346-a990-496e-56d5-210dcfc00880 BootID:72c5fd24-1fe8-4a73-8b41-5591877dd033 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:cc:70:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:cc:70:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:dd:23:d8:4a:3f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 13:08:34.108611 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.108610 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 13:08:34.108728 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.108716 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 13:08:34.111885 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.111863 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 13:08:34.112031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.111889 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 13:08:34.112733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.112724 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 13:08:34.112769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.112736 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 13:08:34.112769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.112743 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:08:34.113548 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.113537 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:08:34.114696 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.114685 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 13:08:34.120069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120056 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:08:34.120107 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120081 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:08:34.120205 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120195 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 13:08:34.120252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120220 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 16 13:08:34.120252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120229 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 13:08:34.120252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120243 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 13:08:34.120252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120251 2581 kubelet.go:398] "Adding apiserver pod source" Sep 16 13:08:34.120415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120260 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 13:08:34.120628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120616 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:08:34.120657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.120637 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:08:34.123072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.123058 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 13:08:34.125846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.125831 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 13:08:34.125897 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.125861 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 13:08:34.127850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127838 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127859 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127866 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127871 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127879 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127885 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127899 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127904 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127911 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127916 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127928 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 13:08:34.127928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.127934 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 13:08:34.128851 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.128840 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 13:08:34.128851 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.128851 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 13:08:34.131027 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.130983 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-73.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 13:08:34.131097 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.131038 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 13:08:34.131097 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.131090 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 13:08:34.132592 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.132580 2581 server.go:1268] "Started kubelet" Sep 16 13:08:34.133071 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.133009 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 13:08:34.133071 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.133035 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 13:08:34.133219 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.133101 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 13:08:34.133872 ip-10-0-133-73 systemd[1]: Started Kubernetes Kubelet. Sep 16 13:08:34.134897 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.134622 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 13:08:34.136771 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.136755 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 16 13:08:34.142600 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.142575 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 13:08:34.144316 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.144301 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vtb4k" Sep 16 13:08:34.144876 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.144853 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 13:08:34.145647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.145632 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 13:08:34.145759 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.145736 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.146377 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146356 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 13:08:34.146451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146401 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 13:08:34.146451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146417 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 13:08:34.146543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146473 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 16 13:08:34.146543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146483 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 16 13:08:34.146646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146629 2581 factory.go:153] Registering CRI-O factory Sep 16 13:08:34.146743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146648 2581 factory.go:223] Registration of the crio container factory successfully Sep 16 13:08:34.146743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146699 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 13:08:34.146743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146708 2581 factory.go:55] Registering systemd factory Sep 16 13:08:34.146743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146715 2581 factory.go:223] Registration of the systemd container factory successfully Sep 16 13:08:34.146743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146730 2581 factory.go:103] Registering Raw factory Sep 16 13:08:34.146743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.146744 2581 manager.go:1196] Started watching for new ooms in manager Sep 16 13:08:34.147136 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.147111 2581 manager.go:349] Starting recovery of all containers Sep 16 13:08:34.153294 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.152834 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vtb4k" Sep 16 13:08:34.153975 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.153949 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 13:08:34.154168 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.154089 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 13:08:34.154261 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.153000 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-73.ec2.internal.18d5cf148da84c89 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-73.ec2.internal,UID:ip-10-0-133-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-73.ec2.internal,},FirstTimestamp:2026-09-16 13:08:34.132552841 +0000 UTC m=+0.397437045,LastTimestamp:2026-09-16 13:08:34.132552841 +0000 UTC m=+0.397437045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-73.ec2.internal,}" Sep 16 13:08:34.156338 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.156321 2581 manager.go:354] Recovery completed Sep 16 13:08:34.160989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.160977 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.162686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.162667 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.162764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.162699 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.162764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.162714 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.163282 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.163269 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 13:08:34.163282 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.163279 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 13:08:34.163390 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.163293 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:08:34.163390 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.163348 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.164031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.164016 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.164103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.164045 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.164103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.164059 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.164572 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.164517 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-73.ec2.internal.18d5cf148f740ec6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-73.ec2.internal,UID:ip-10-0-133-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-73.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-73.ec2.internal,},FirstTimestamp:2026-09-16 13:08:34.16268359 +0000 UTC m=+0.427567797,LastTimestamp:2026-09-16 13:08:34.16268359 +0000 UTC m=+0.427567797,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-73.ec2.internal,}" Sep 16 13:08:34.165395 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.165384 2581 policy_none.go:49] "None policy: Start" Sep 16 13:08:34.165433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.165398 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 13:08:34.165433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.165409 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 13:08:34.167380 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.167365 2581 policy_none.go:47] "Start" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.200686 2581 manager.go:337] "Starting Device Plugin manager" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.200836 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.200849 2581 server.go:87] "Starting device plugin registration server" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.201115 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.201141 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.201236 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.201322 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.201333 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.201898 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 13:08:34.217689 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.201945 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.233671 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.233649 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 13:08:34.234850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.234833 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 13:08:34.234939 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.234879 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 13:08:34.234939 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.234901 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 13:08:34.235026 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.234964 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 13:08:34.237398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.237380 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:08:34.301681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.301658 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.303954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.303939 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.304037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.303969 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.304037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.303984 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.304037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.304005 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.310314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.310252 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.310314 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.310271 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-73.ec2.internal\": node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.324616 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.324599 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.335817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.335787 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal"] Sep 16 13:08:34.335915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.335861 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.337175 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.337161 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.337235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.337189 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.337235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.337203 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.339652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.339638 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.339747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.339733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.339802 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.339760 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.340384 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.340369 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.340456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.340396 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.340456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.340406 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.340917 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.340897 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.340995 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.340926 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.340995 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.340940 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.343281 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.343266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.343375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.343288 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:08:34.343905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.343890 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:08:34.343976 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.343916 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:08:34.343976 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.343926 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:08:34.347533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.347518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af97f616135893b3ca8cfcca10b953ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal\" (UID: \"af97f616135893b3ca8cfcca10b953ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.347587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.347541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3467b75dabbe10a15451a6223ddb8b9c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-73.ec2.internal\" (UID: \"3467b75dabbe10a15451a6223ddb8b9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.347587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.347558 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af97f616135893b3ca8cfcca10b953ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal\" (UID: \"af97f616135893b3ca8cfcca10b953ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.365726 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.365709 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-73.ec2.internal\" not found" node="ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.370222 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.370209 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-73.ec2.internal\" not found" node="ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.425193 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.425172 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.447667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.447644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3467b75dabbe10a15451a6223ddb8b9c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-73.ec2.internal\" (UID: \"3467b75dabbe10a15451a6223ddb8b9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.447739 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.447670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af97f616135893b3ca8cfcca10b953ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal\" (UID: \"af97f616135893b3ca8cfcca10b953ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.447739 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.447686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af97f616135893b3ca8cfcca10b953ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal\" (UID: \"af97f616135893b3ca8cfcca10b953ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.447806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.447745 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af97f616135893b3ca8cfcca10b953ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal\" (UID: \"af97f616135893b3ca8cfcca10b953ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.447806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.447764 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af97f616135893b3ca8cfcca10b953ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal\" (UID: \"af97f616135893b3ca8cfcca10b953ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.447806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.447745 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3467b75dabbe10a15451a6223ddb8b9c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-73.ec2.internal\" (UID: \"3467b75dabbe10a15451a6223ddb8b9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.525245 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.525221 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.626291 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.626241 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:34.668663 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.668638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.674111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:34.673520 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" Sep 16 13:08:34.727328 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:34.727301 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:35.972901 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:35.972875 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:36.073426 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.073373 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-73.ec2.internal\" not found" Sep 16 13:08:36.098483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.098465 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:08:36.191631 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.191607 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" Sep 16 13:08:36.202983 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.202955 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:08:36.204360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.204345 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" Sep 16 13:08:36.215451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.215432 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:08:36.225578 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.225559 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 13:08:36.225732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.225707 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:08:36.225785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.225724 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:08:36.265883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.265854 2581 apiserver.go:52] "Watching apiserver" Sep 16 13:08:36.283033 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.283011 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 13:08:36.284785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.284756 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4ch6p","openshift-ovn-kubernetes/ovnkube-node-w7mng","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874","openshift-dns/node-resolver-m2lwv","openshift-image-registry/node-ca-lv88k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal","openshift-multus/multus-5k5jw","openshift-multus/multus-additional-cni-plugins-bmbbz","openshift-multus/network-metrics-daemon-zs895","openshift-network-diagnostics/network-check-target-f4nw4","kube-system/global-pull-secret-syncer-zvh5v","kube-system/konnectivity-agent-hpj8d","kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9cgfl"] Sep 16 13:08:36.286897 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.286879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.289322 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.289295 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 13:08:36.289597 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.289582 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.289704 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.289683 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.289770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.289711 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x882g\"" Sep 16 13:08:36.289770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.289740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.289870 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.289805 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 13:08:36.289945 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.289871 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:36.290329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.290308 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 13:08:36.291230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.291215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.292962 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.292946 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxxbq\"" Sep 16 13:08:36.293206 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.293192 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.293312 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.293297 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 13:08:36.293374 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.293347 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.293986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.293970 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.295686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295673 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m768p\"" Sep 16 13:08:36.295763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295754 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.295841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-conf-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.295917 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6dc3916-9abe-4021-a871-0deca7e008f1-kubelet-config\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.295917 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-socket-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.295917 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295907 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzg6k\" (UniqueName: \"kubernetes.io/projected/57862b11-5908-4bbd-bad9-164a84fce95c-kube-api-access-hzg6k\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295956 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295953 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-cni-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295979 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-k8s-cni-cncf-io\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.295992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6dc3916-9abe-4021-a871-0deca7e008f1-dbus\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.296065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-hosts-file\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvw8w\" (UniqueName: \"kubernetes.io/projected/1d00521c-a61f-45d7-8f29-68d7b45a82ea-kube-api-access-qvw8w\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-device-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-system-cni-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296131 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-cni-multus\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-etc-kubernetes\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-tmp-dir\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296176 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-sys-fs\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-registration-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-socket-dir-parent\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296219 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-cni-bin\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-multus-certs\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296274 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d00521c-a61f-45d7-8f29-68d7b45a82ea-cni-binary-copy\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-netns\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-kubelet\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-hostroot\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296441 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-cnibin\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-daemon-config\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.296899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.296497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjwm\" (UniqueName: \"kubernetes.io/projected/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-kube-api-access-4cjwm\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.298070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.298054 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mcv2\"" Sep 16 13:08:36.298158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.298126 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 13:08:36.298363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.298347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.298435 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.298363 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.298525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.298453 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.300319 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.300300 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rjjxt\"" Sep 16 13:08:36.300409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.300306 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.300409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.300357 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.300587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.300571 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:36.300660 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.300639 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:36.302420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.302403 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:08:36.302781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.302766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.304884 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.304868 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 13:08:36.304971 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.304931 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 13:08:36.304971 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.304939 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2mtf\"" Sep 16 13:08:36.305078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.304975 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:03:35 +0000 UTC" deadline="2028-04-23 01:04:42.035157312 +0000 UTC" Sep 16 13:08:36.305078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.305018 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14027h56m5.730141173s" Sep 16 13:08:36.305477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.305462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.307591 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.307574 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 13:08:36.307682 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.307646 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:36.307738 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.307698 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:36.308042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.307785 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7v5fn\"" Sep 16 13:08:36.308042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.307839 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.308042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.307915 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.308042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.307920 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 13:08:36.308846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.308827 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 13:08:36.308944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.308926 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 13:08:36.310111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.310096 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.312370 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.312355 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 13:08:36.312446 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.312359 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:08:36.312564 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.312552 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nnp7v\"" Sep 16 13:08:36.312614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.312584 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 13:08:36.312646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.312622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.314977 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.314959 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 13:08:36.315067 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.315053 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s7426\"" Sep 16 13:08:36.315251 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.315239 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 13:08:36.325305 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.325247 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vqx57" Sep 16 13:08:36.334046 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.334030 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vqx57" Sep 16 13:08:36.356537 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.356499 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3467b75dabbe10a15451a6223ddb8b9c.slice/crio-504f20524005a019aa907f6919b68c5adeafc3c61b91743d76ce1e2c5f9cea7b WatchSource:0}: Error finding container 504f20524005a019aa907f6919b68c5adeafc3c61b91743d76ce1e2c5f9cea7b: Status 404 returned error can't find the container with id 504f20524005a019aa907f6919b68c5adeafc3c61b91743d76ce1e2c5f9cea7b Sep 16 13:08:36.356732 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.356711 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf97f616135893b3ca8cfcca10b953ab.slice/crio-0b33cf009dde2cc3593b47c62ec0bb1be77db62ba906de15e564bea6499ef737 WatchSource:0}: Error finding container 0b33cf009dde2cc3593b47c62ec0bb1be77db62ba906de15e564bea6499ef737: Status 404 returned error can't find the container with id 0b33cf009dde2cc3593b47c62ec0bb1be77db62ba906de15e564bea6499ef737 Sep 16 13:08:36.360641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.360626 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:08:36.383192 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.383151 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" event={"ID":"af97f616135893b3ca8cfcca10b953ab","Type":"ContainerStarted","Data":"0b33cf009dde2cc3593b47c62ec0bb1be77db62ba906de15e564bea6499ef737"} Sep 16 13:08:36.384243 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.384219 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" event={"ID":"3467b75dabbe10a15451a6223ddb8b9c","Type":"ContainerStarted","Data":"504f20524005a019aa907f6919b68c5adeafc3c61b91743d76ce1e2c5f9cea7b"} Sep 16 13:08:36.392562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.392541 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 13:08:36.397298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-slash\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.397362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-node-log\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.397362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397319 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-log-socket\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.397362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6dc3916-9abe-4021-a871-0deca7e008f1-kubelet-config\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.397469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-socket-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.397469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397420 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6dc3916-9abe-4021-a871-0deca7e008f1-kubelet-config\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.397469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzg6k\" (UniqueName: \"kubernetes.io/projected/57862b11-5908-4bbd-bad9-164a84fce95c-kube-api-access-hzg6k\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.397469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysctl-d\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397473 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-socket-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-cni-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397509 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-k8s-cni-cncf-io\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-modprobe-d\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397525 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-cni-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-systemd\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397578 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-k8s-cni-cncf-io\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397582 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89116514-29df-4cd2-8df1-f5a946929c67-tmp\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.397640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397626 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397667 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-system-cni-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-cni-multus\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6366a1ee-e8d7-404f-92c1-99b274f32385-iptables-alerter-script\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-system-cni-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-sys\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397757 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-cni-multus\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-systemd\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg52t\" (UniqueName: \"kubernetes.io/projected/906cc307-f99a-4be6-a72b-38024d094c15-kube-api-access-jg52t\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-cni-bin\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbh2\" (UniqueName: \"kubernetes.io/projected/75bd51a9-a796-46f3-ae52-fb1b49353736-kube-api-access-hhbh2\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-kubernetes\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-run-netns\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.398009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-socket-dir-parent\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-cni-bin\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.397999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysctl-conf\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398018 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-socket-dir-parent\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-host\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-cni-bin\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-etc-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovnkube-script-lib\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d00521c-a61f-45d7-8f29-68d7b45a82ea-cni-binary-copy\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398123 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-kubelet\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-hostroot\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ef40846-8a2e-4db0-9b5c-7a0f68d62b40-konnectivity-ca\") pod \"konnectivity-agent-hpj8d\" (UID: \"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40\") " pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-cnibin\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ef40846-8a2e-4db0-9b5c-7a0f68d62b40-agent-certs\") pod \"konnectivity-agent-hpj8d\" (UID: \"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40\") " pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-ovn\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.398567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-hostroot\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovn-node-metrics-cert\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398248 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-var-lib-kubelet\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-cnibin\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhgvv\" (UniqueName: \"kubernetes.io/projected/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-kube-api-access-jhgvv\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00be8b34-1abf-4801-b955-ba483d5f6c34-serviceca\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-kubelet\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-conf-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-conf-dir\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89116514-29df-4cd2-8df1-f5a946929c67-etc-tuned\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-system-cni-dir\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398522 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398527 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d00521c-a61f-45d7-8f29-68d7b45a82ea-cni-binary-copy\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398531 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6dc3916-9abe-4021-a871-0deca7e008f1-dbus\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.399343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398592 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6dc3916-9abe-4021-a871-0deca7e008f1-dbus\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-hosts-file\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398644 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398687 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-hosts-file\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.398687 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvw8w\" (UniqueName: \"kubernetes.io/projected/1d00521c-a61f-45d7-8f29-68d7b45a82ea-kube-api-access-qvw8w\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-device-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-etc-kubernetes\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.398758 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:36.898736812 +0000 UTC m=+2.018463351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-var-lib-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398778 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-etc-kubernetes\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398797 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-env-overrides\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkms\" (UniqueName: \"kubernetes.io/projected/00be8b34-1abf-4801-b955-ba483d5f6c34-kube-api-access-7zkms\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-cnibin\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-tmp-dir\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.399960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398864 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-device-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nh9\" (UniqueName: \"kubernetes.io/projected/6366a1ee-e8d7-404f-92c1-99b274f32385-kube-api-access-k4nh9\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysconfig\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-run\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.398978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-cni-netd\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-sys-fs\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399075 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-sys-fs\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399085 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-tmp-dir\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6366a1ee-e8d7-404f-92c1-99b274f32385-host-slash\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-lib-modules\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggqd\" (UniqueName: \"kubernetes.io/projected/89116514-29df-4cd2-8df1-f5a946929c67-kube-api-access-sggqd\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399173 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovnkube-config\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-registration-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399227 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57862b11-5908-4bbd-bad9-164a84fce95c-registration-dir\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-multus-certs\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399265 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-systemd-units\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00be8b34-1abf-4801-b955-ba483d5f6c34-host\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.400445 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399333 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-multus-certs\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-netns\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399459 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d00521c-a61f-45d7-8f29-68d7b45a82ea-host-run-netns\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-daemon-config\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjwm\" (UniqueName: \"kubernetes.io/projected/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-kube-api-access-4cjwm\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.399512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-var-lib-kubelet\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.400914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.400428 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d00521c-a61f-45d7-8f29-68d7b45a82ea-multus-daemon-config\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.408472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.408454 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 13:08:36.414141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.413217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvw8w\" (UniqueName: \"kubernetes.io/projected/1d00521c-a61f-45d7-8f29-68d7b45a82ea-kube-api-access-qvw8w\") pod \"multus-5k5jw\" (UID: \"1d00521c-a61f-45d7-8f29-68d7b45a82ea\") " pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.414141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.413229 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjwm\" (UniqueName: \"kubernetes.io/projected/d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece-kube-api-access-4cjwm\") pod \"node-resolver-m2lwv\" (UID: \"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece\") " pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.414141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.413225 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzg6k\" (UniqueName: \"kubernetes.io/projected/57862b11-5908-4bbd-bad9-164a84fce95c-kube-api-access-hzg6k\") pod \"aws-ebs-csi-driver-node-hj874\" (UID: \"57862b11-5908-4bbd-bad9-164a84fce95c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.500418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ef40846-8a2e-4db0-9b5c-7a0f68d62b40-konnectivity-ca\") pod \"konnectivity-agent-hpj8d\" (UID: \"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40\") " pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.500522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ef40846-8a2e-4db0-9b5c-7a0f68d62b40-agent-certs\") pod \"konnectivity-agent-hpj8d\" (UID: \"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40\") " pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.500522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-ovn\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovn-node-metrics-cert\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgvv\" (UniqueName: \"kubernetes.io/projected/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-kube-api-access-jhgvv\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500544 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-ovn\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00be8b34-1abf-4801-b955-ba483d5f6c34-serviceca\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.500715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.500715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-kubelet\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500728 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-kubelet\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89116514-29df-4cd2-8df1-f5a946929c67-etc-tuned\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.500908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-system-cni-dir\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.500908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.500908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.500908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-var-lib-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-env-overrides\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500921 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-system-cni-dir\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500937 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ef40846-8a2e-4db0-9b5c-7a0f68d62b40-konnectivity-ca\") pod \"konnectivity-agent-hpj8d\" (UID: \"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40\") " pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkms\" (UniqueName: \"kubernetes.io/projected/00be8b34-1abf-4801-b955-ba483d5f6c34-kube-api-access-7zkms\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-cnibin\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nh9\" (UniqueName: \"kubernetes.io/projected/6366a1ee-e8d7-404f-92c1-99b274f32385-kube-api-access-k4nh9\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.500994 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501080 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00be8b34-1abf-4801-b955-ba483d5f6c34-serviceca\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501098 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysconfig\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-run\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501179 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75bd51a9-a796-46f3-ae52-fb1b49353736-cnibin\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.501189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-cni-netd\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501220 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6366a1ee-e8d7-404f-92c1-99b274f32385-host-slash\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501228 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-var-lib-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501236 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501251 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-run\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-lib-modules\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysconfig\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501315 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-cni-netd\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggqd\" (UniqueName: \"kubernetes.io/projected/89116514-29df-4cd2-8df1-f5a946929c67-kube-api-access-sggqd\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501338 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6366a1ee-e8d7-404f-92c1-99b274f32385-host-slash\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovnkube-config\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-systemd-units\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501386 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-lib-modules\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00be8b34-1abf-4801-b955-ba483d5f6c34-host\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501476 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.501781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-var-lib-kubelet\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501511 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-systemd-units\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501541 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00be8b34-1abf-4801-b955-ba483d5f6c34-host\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501543 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-env-overrides\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501547 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501546 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-var-lib-kubelet\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-slash\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-node-log\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501669 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-slash\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-node-log\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-log-socket\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysctl-d\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501745 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-log-socket\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-modprobe-d\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-systemd\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89116514-29df-4cd2-8df1-f5a946929c67-tmp\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501827 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-modprobe-d\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:36.502555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501856 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysctl-d\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6366a1ee-e8d7-404f-92c1-99b274f32385-iptables-alerter-script\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501877 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-systemd\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovnkube-config\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501907 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-sys\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-systemd\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.501937 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg52t\" (UniqueName: \"kubernetes.io/projected/906cc307-f99a-4be6-a72b-38024d094c15-kube-api-access-jg52t\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.501976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-cni-bin\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.501996 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:37.001978707 +0000 UTC m=+2.121705249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502010 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-cni-bin\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502060 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-sys\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbh2\" (UniqueName: \"kubernetes.io/projected/75bd51a9-a796-46f3-ae52-fb1b49353736-kube-api-access-hhbh2\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502095 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-run-systemd\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-kubernetes\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.503432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-run-netns\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-kubernetes\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502187 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysctl-conf\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-host\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-etc-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovnkube-script-lib\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502495 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-host-run-netns\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502521 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-etc-sysctl-conf\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502580 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-etc-openvswitch\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502685 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89116514-29df-4cd2-8df1-f5a946929c67-host\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502814 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovnkube-script-lib\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.502991 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75bd51a9-a796-46f3-ae52-fb1b49353736-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.503041 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6366a1ee-e8d7-404f-92c1-99b274f32385-iptables-alerter-script\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.503263 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-ovn-node-metrics-cert\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.503343 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89116514-29df-4cd2-8df1-f5a946929c67-etc-tuned\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.504235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.503576 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ef40846-8a2e-4db0-9b5c-7a0f68d62b40-agent-certs\") pod \"konnectivity-agent-hpj8d\" (UID: \"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40\") " pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.504899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.504756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89116514-29df-4cd2-8df1-f5a946929c67-tmp\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.514625 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.514603 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:08:36.514729 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.514629 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:08:36.514729 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.514644 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:36.514729 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.514719 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:37.014699509 +0000 UTC m=+2.134426055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:36.515119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.515092 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggqd\" (UniqueName: \"kubernetes.io/projected/89116514-29df-4cd2-8df1-f5a946929c67-kube-api-access-sggqd\") pod \"tuned-9cgfl\" (UID: \"89116514-29df-4cd2-8df1-f5a946929c67\") " pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.515411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.515386 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkms\" (UniqueName: \"kubernetes.io/projected/00be8b34-1abf-4801-b955-ba483d5f6c34-kube-api-access-7zkms\") pod \"node-ca-lv88k\" (UID: \"00be8b34-1abf-4801-b955-ba483d5f6c34\") " pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.515543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.515526 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nh9\" (UniqueName: \"kubernetes.io/projected/6366a1ee-e8d7-404f-92c1-99b274f32385-kube-api-access-k4nh9\") pod \"iptables-alerter-4ch6p\" (UID: \"6366a1ee-e8d7-404f-92c1-99b274f32385\") " pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.515606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.515555 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgvv\" (UniqueName: \"kubernetes.io/projected/0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81-kube-api-access-jhgvv\") pod \"ovnkube-node-w7mng\" (UID: \"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.516638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.516622 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg52t\" (UniqueName: \"kubernetes.io/projected/906cc307-f99a-4be6-a72b-38024d094c15-kube-api-access-jg52t\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:36.516963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.516947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbh2\" (UniqueName: \"kubernetes.io/projected/75bd51a9-a796-46f3-ae52-fb1b49353736-kube-api-access-hhbh2\") pod \"multus-additional-cni-plugins-bmbbz\" (UID: \"75bd51a9-a796-46f3-ae52-fb1b49353736\") " pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.617650 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.617583 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5k5jw" Sep 16 13:08:36.628168 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.628144 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d00521c_a61f_45d7_8f29_68d7b45a82ea.slice/crio-b3b68ac159b460a8f068ce93fd013f03585090b766ea862a0578a606d6df7666 WatchSource:0}: Error finding container b3b68ac159b460a8f068ce93fd013f03585090b766ea862a0578a606d6df7666: Status 404 returned error can't find the container with id b3b68ac159b460a8f068ce93fd013f03585090b766ea862a0578a606d6df7666 Sep 16 13:08:36.636494 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.636476 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" Sep 16 13:08:36.644531 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.644511 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57862b11_5908_4bbd_bad9_164a84fce95c.slice/crio-71c00e473545d91f1d8b384cf63207a0d92a35517a5caf4577305e4daad4fb98 WatchSource:0}: Error finding container 71c00e473545d91f1d8b384cf63207a0d92a35517a5caf4577305e4daad4fb98: Status 404 returned error can't find the container with id 71c00e473545d91f1d8b384cf63207a0d92a35517a5caf4577305e4daad4fb98 Sep 16 13:08:36.664330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.664309 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m2lwv" Sep 16 13:08:36.668129 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.668110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lv88k" Sep 16 13:08:36.672888 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.672861 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ec3cf3_f7e9_416b_9a45_2faf2cd25ece.slice/crio-35ec201cd1d4b3c54a82e623a035ef4aee35b0740fb82753a5fb9b3f4669770d WatchSource:0}: Error finding container 35ec201cd1d4b3c54a82e623a035ef4aee35b0740fb82753a5fb9b3f4669770d: Status 404 returned error can't find the container with id 35ec201cd1d4b3c54a82e623a035ef4aee35b0740fb82753a5fb9b3f4669770d Sep 16 13:08:36.672977 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.672948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" Sep 16 13:08:36.676776 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.676756 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00be8b34_1abf_4801_b955_ba483d5f6c34.slice/crio-d73f0d75292662dbf720d4a791e6828c8873586d02a0a789c92a05a4962d5fc0 WatchSource:0}: Error finding container d73f0d75292662dbf720d4a791e6828c8873586d02a0a789c92a05a4962d5fc0: Status 404 returned error can't find the container with id d73f0d75292662dbf720d4a791e6828c8873586d02a0a789c92a05a4962d5fc0 Sep 16 13:08:36.678608 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.678588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" Sep 16 13:08:36.682774 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.682753 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89116514_29df_4cd2_8df1_f5a946929c67.slice/crio-6da5e6fbd5005e5975b0125d7519d44cb869869587460d3f3c641455a1acb328 WatchSource:0}: Error finding container 6da5e6fbd5005e5975b0125d7519d44cb869869587460d3f3c641455a1acb328: Status 404 returned error can't find the container with id 6da5e6fbd5005e5975b0125d7519d44cb869869587460d3f3c641455a1acb328 Sep 16 13:08:36.684467 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.684453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:08:36.687914 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.687883 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bd51a9_a796_46f3_ae52_fb1b49353736.slice/crio-5b90cdeb4912cee3dfb77b4680670e194645a5c41aed7c4f12562d8ebcc78341 WatchSource:0}: Error finding container 5b90cdeb4912cee3dfb77b4680670e194645a5c41aed7c4f12562d8ebcc78341: Status 404 returned error can't find the container with id 5b90cdeb4912cee3dfb77b4680670e194645a5c41aed7c4f12562d8ebcc78341 Sep 16 13:08:36.690114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.690093 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4ch6p" Sep 16 13:08:36.692915 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.692896 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfe9ac2_4ce9_4ff9_957c_a6fc6e308e81.slice/crio-2a5f029fb21ba57d752b2a810b726ee4e74d7482506edb180f54d2ef8766e440 WatchSource:0}: Error finding container 2a5f029fb21ba57d752b2a810b726ee4e74d7482506edb180f54d2ef8766e440: Status 404 returned error can't find the container with id 2a5f029fb21ba57d752b2a810b726ee4e74d7482506edb180f54d2ef8766e440 Sep 16 13:08:36.694954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.694475 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:36.698672 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.698651 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6366a1ee_e8d7_404f_92c1_99b274f32385.slice/crio-c08fb74d9027392e85f1f3ccf147ce11cf5c618f668c02940fe4ea086c298f77 WatchSource:0}: Error finding container c08fb74d9027392e85f1f3ccf147ce11cf5c618f668c02940fe4ea086c298f77: Status 404 returned error can't find the container with id c08fb74d9027392e85f1f3ccf147ce11cf5c618f668c02940fe4ea086c298f77 Sep 16 13:08:36.703093 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:08:36.703075 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef40846_8a2e_4db0_9b5c_7a0f68d62b40.slice/crio-3fdf435abb174276a86d17de71eb56b57c9d3e241d098c288c2b4d39c904e26b WatchSource:0}: Error finding container 3fdf435abb174276a86d17de71eb56b57c9d3e241d098c288c2b4d39c904e26b: Status 404 returned error can't find the container with id 3fdf435abb174276a86d17de71eb56b57c9d3e241d098c288c2b4d39c904e26b Sep 16 13:08:36.833567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.833524 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:08:36.899035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.898948 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:08:36.905198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:36.905171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:36.905366 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.905304 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:36.905424 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:36.905368 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:37.905349732 +0000 UTC m=+3.025076274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:37.005762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.005731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:37.005933 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.005878 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:37.005994 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.005941 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:38.00592258 +0000 UTC m=+3.125649126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:37.052897 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.052866 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:08:37.106866 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.106828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:37.107068 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.107048 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:08:37.107138 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.107077 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:08:37.107138 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.107090 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:37.107241 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.107145 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:38.107127407 +0000 UTC m=+3.226853936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:37.337229 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.337124 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:03:36 +0000 UTC" deadline="2028-05-23 14:52:31.788122496 +0000 UTC" Sep 16 13:08:37.337229 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.337184 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14761h43m54.450942027s" Sep 16 13:08:37.382475 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.382442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:37.382638 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.382574 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:37.389752 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.389703 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lv88k" event={"ID":"00be8b34-1abf-4801-b955-ba483d5f6c34","Type":"ContainerStarted","Data":"d73f0d75292662dbf720d4a791e6828c8873586d02a0a789c92a05a4962d5fc0"} Sep 16 13:08:37.392786 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.392756 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" event={"ID":"57862b11-5908-4bbd-bad9-164a84fce95c","Type":"ContainerStarted","Data":"71c00e473545d91f1d8b384cf63207a0d92a35517a5caf4577305e4daad4fb98"} Sep 16 13:08:37.398765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.398738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hpj8d" event={"ID":"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40","Type":"ContainerStarted","Data":"3fdf435abb174276a86d17de71eb56b57c9d3e241d098c288c2b4d39c904e26b"} Sep 16 13:08:37.410203 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.406373 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerStarted","Data":"5b90cdeb4912cee3dfb77b4680670e194645a5c41aed7c4f12562d8ebcc78341"} Sep 16 13:08:37.410203 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.408155 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m2lwv" event={"ID":"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece","Type":"ContainerStarted","Data":"35ec201cd1d4b3c54a82e623a035ef4aee35b0740fb82753a5fb9b3f4669770d"} Sep 16 13:08:37.410581 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.410536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5k5jw" event={"ID":"1d00521c-a61f-45d7-8f29-68d7b45a82ea","Type":"ContainerStarted","Data":"b3b68ac159b460a8f068ce93fd013f03585090b766ea862a0578a606d6df7666"} Sep 16 13:08:37.428305 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.428267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4ch6p" event={"ID":"6366a1ee-e8d7-404f-92c1-99b274f32385","Type":"ContainerStarted","Data":"c08fb74d9027392e85f1f3ccf147ce11cf5c618f668c02940fe4ea086c298f77"} Sep 16 13:08:37.429915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.429891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"2a5f029fb21ba57d752b2a810b726ee4e74d7482506edb180f54d2ef8766e440"} Sep 16 13:08:37.431433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.431408 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" event={"ID":"89116514-29df-4cd2-8df1-f5a946929c67","Type":"ContainerStarted","Data":"6da5e6fbd5005e5975b0125d7519d44cb869869587460d3f3c641455a1acb328"} Sep 16 13:08:37.913053 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:37.913010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:37.913236 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.913153 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:37.913236 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:37.913221 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:39.913204117 +0000 UTC m=+5.032930648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:38.014138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.014063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:38.014339 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.014234 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:38.014339 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.014316 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:40.014295689 +0000 UTC m=+5.134022232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:38.115607 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.115337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:38.115607 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.115495 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:08:38.115607 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.115519 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:08:38.115607 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.115532 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:38.115607 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.115598 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:40.115579496 +0000 UTC m=+5.235306040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:38.338393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.338299 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:03:36 +0000 UTC" deadline="2028-03-28 13:45:55.788481374 +0000 UTC" Sep 16 13:08:38.338393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.338339 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13416h37m17.450146174s" Sep 16 13:08:38.381102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.380422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:38.381102 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.380551 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:38.381102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.380927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:38.381102 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:38.381020 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:38.532549 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:38.532516 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:08:39.383974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:39.383414 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:39.383974 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:39.383595 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:39.930118 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:39.930075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:39.930322 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:39.930238 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:39.930438 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:39.930330 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:43.930308321 +0000 UTC m=+9.050034903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:40.030767 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:40.030731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:40.030929 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.030896 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:40.031007 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.030975 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:44.030946888 +0000 UTC m=+9.150673435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:40.131738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:40.131703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:40.131921 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.131902 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:08:40.131982 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.131930 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:08:40.131982 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.131944 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:40.132084 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.132016 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:44.131996366 +0000 UTC m=+9.251722899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:40.381213 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:40.381180 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:40.381383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:40.381181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:40.381383 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.381315 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:40.381499 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:40.381433 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:41.382215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:41.382182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:41.382686 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:41.382332 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:42.380825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:42.380784 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:42.381023 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:42.380801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:42.381023 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:42.380930 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:42.381148 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:42.381022 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:43.380524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:43.380479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:43.381047 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:43.380612 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:43.966964 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:43.966250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:43.966964 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:43.966476 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:43.966964 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:43.966641 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:51.966621798 +0000 UTC m=+17.086348345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:44.067898 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:44.067850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:44.068069 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.068026 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:44.068127 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.068094 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:52.068075421 +0000 UTC m=+17.187802169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:44.169295 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:44.169143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:44.169443 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.169302 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:08:44.169443 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.169324 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:08:44.169443 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.169335 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:44.169443 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.169404 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:08:52.169377438 +0000 UTC m=+17.289103979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:44.381228 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:44.381189 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:44.381636 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.381320 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:44.381636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:44.381189 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:44.381636 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:44.381413 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:45.381492 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:45.381461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:45.381920 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:45.381570 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:46.381247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:46.381188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:46.381481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:46.381341 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:46.381481 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:46.381449 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:46.381481 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:46.381337 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:47.380951 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:47.380909 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:47.381125 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:47.381036 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:48.381174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:48.381137 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:48.381620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:48.381148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:48.381620 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:48.381260 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:48.381620 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:48.381345 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:49.381069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:49.381038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:49.381240 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:49.381137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:50.380602 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:50.380568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:50.380602 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:50.380599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:50.380854 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:50.380685 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:50.380854 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:50.380819 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:51.381105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:51.381073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:51.381482 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:51.381174 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:52.022483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:52.022447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:52.022702 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.022589 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:52.022702 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.022656 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:08.022637722 +0000 UTC m=+33.142364271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:08:52.123635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:52.123592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:52.123808 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.123722 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:52.123866 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.123824 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:08.123801307 +0000 UTC m=+33.243527858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:08:52.224005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:52.223951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:52.224191 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.224131 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:08:52.224191 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.224158 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:08:52.224191 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.224171 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:52.224378 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.224238 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:08.22422043 +0000 UTC m=+33.343946957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:08:52.380339 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:52.380252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:52.380501 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:52.380253 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:52.380501 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.380376 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:52.380501 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:52.380473 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:53.380382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:53.380348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:53.380853 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:53.380459 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:54.381048 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:54.381018 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:54.381048 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:54.381044 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:54.381571 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:54.381147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:54.381571 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:54.381304 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:55.381926 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:55.381895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:55.382255 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:55.381973 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:56.380972 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.380758 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:56.381168 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:56.381106 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:56.381168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.380799 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:56.381301 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:56.381188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:56.482624 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.482592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" event={"ID":"3467b75dabbe10a15451a6223ddb8b9c","Type":"ContainerStarted","Data":"829220e200db5a55c4ec64ea7531e08b0150a5533d4ebb976f801b4b25234414"} Sep 16 13:08:56.485788 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.485769 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:08:56.486113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.486085 2581 generic.go:356] "Generic (PLEG): container finished" podID="0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81" containerID="2f3e82a27d13a75f41da402f2ffcf49c68cc849c10bfe23f0913d1c3a8640e41" exitCode=1 Sep 16 13:08:56.486212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.486162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"a98516fd7b292ba3263377648c8b2dc331ccac6b48f382c5428666f6ebff1fd7"} Sep 16 13:08:56.486290 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.486227 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"1f4163ac644d074bf084dc211d672ffbf746992001ea75f09e7399a48057c8b0"} Sep 16 13:08:56.486352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.486274 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerDied","Data":"2f3e82a27d13a75f41da402f2ffcf49c68cc849c10bfe23f0913d1c3a8640e41"} Sep 16 13:08:56.486352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.486325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"1cc4b8a5a5aefbd966eb2e543ae2350d1e34c864a6993436d3acc98cbc72c06f"} Sep 16 13:08:56.487819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.487796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" event={"ID":"89116514-29df-4cd2-8df1-f5a946929c67","Type":"ContainerStarted","Data":"9125b3e6d720886f33cafc683eafb1c5f8b2164afe05e571c249808ab420fa0f"} Sep 16 13:08:56.489336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.489314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5k5jw" event={"ID":"1d00521c-a61f-45d7-8f29-68d7b45a82ea","Type":"ContainerStarted","Data":"cf0b9a97ced7f045f546c9c59b280d08419ee2f3b4fb75db251b5093cc965367"} Sep 16 13:08:56.497476 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.497430 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-73.ec2.internal" podStartSLOduration=20.497415814 podStartE2EDuration="20.497415814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:56.497400661 +0000 UTC m=+21.617127238" watchObservedRunningTime="2026-09-16 13:08:56.497415814 +0000 UTC m=+21.617142364" Sep 16 13:08:56.512112 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.512080 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9cgfl" podStartSLOduration=2.582940803 podStartE2EDuration="21.512069647s" totalImagesPullingTime="18.929128844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:56.511533261 +0000 UTC m=+21.631259809" watchObservedRunningTime="2026-09-16 13:08:56.512069647 +0000 UTC m=+21.631796195" Sep 16 13:08:56.527959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:56.527926 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5k5jw" podStartSLOduration=1.9830109409999999 podStartE2EDuration="21.52791648s" totalImagesPullingTime="19.544905539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:56.527596448 +0000 UTC m=+21.647323007" watchObservedRunningTime="2026-09-16 13:08:56.52791648 +0000 UTC m=+21.647643029" Sep 16 13:08:57.381152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.380943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:57.381310 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:57.381186 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:57.492320 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.492273 2581 generic.go:356] "Generic (PLEG): container finished" podID="af97f616135893b3ca8cfcca10b953ab" containerID="977d06a4c9db89c27513af831604bbf03bea9ac8208cb7653d21992b0bf11c64" exitCode=0 Sep 16 13:08:57.492320 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.492316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" event={"ID":"af97f616135893b3ca8cfcca10b953ab","Type":"ContainerDied","Data":"977d06a4c9db89c27513af831604bbf03bea9ac8208cb7653d21992b0bf11c64"} Sep 16 13:08:57.493904 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.493880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4ch6p" event={"ID":"6366a1ee-e8d7-404f-92c1-99b274f32385","Type":"ContainerStarted","Data":"5005a6fd08f60e92f8aa54f59d9194e57f2a25c9fb0ecf4d409a82427e58b9d8"} Sep 16 13:08:57.496532 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.496515 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:08:57.496878 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.496861 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"24bd6bf6b525db29f885ac8fbfd46a07d2f629962b2d16b36b2d65a871afbb20"} Sep 16 13:08:57.496947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.496913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"912648124c6e34c083ba831a8b7c001b13338c7707094e7c741d89e22ae4a66b"} Sep 16 13:08:57.498270 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.498245 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lv88k" event={"ID":"00be8b34-1abf-4801-b955-ba483d5f6c34","Type":"ContainerStarted","Data":"c9a6fa494c5deb590f471eedd148a1568d21d76b422e1d214050431db3395edc"} Sep 16 13:08:57.499687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.499664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" event={"ID":"57862b11-5908-4bbd-bad9-164a84fce95c","Type":"ContainerStarted","Data":"d53865a4d5ec78eb3f7fe21222e7637dbae3bde3267415d03dfbdbad37cb1665"} Sep 16 13:08:57.501001 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.500982 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hpj8d" event={"ID":"9ef40846-8a2e-4db0-9b5c-7a0f68d62b40","Type":"ContainerStarted","Data":"4beaf5183dfacd5be6070ed5c72c8c02ee7c629a5059129aa0676cc1ac0c3921"} Sep 16 13:08:57.502375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.502352 2581 generic.go:356] "Generic (PLEG): container finished" podID="75bd51a9-a796-46f3-ae52-fb1b49353736" containerID="74a4737778f637202684037dd574d595187deca3de14c8210e5c59c3fb647a82" exitCode=0 Sep 16 13:08:57.502448 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.502414 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerDied","Data":"74a4737778f637202684037dd574d595187deca3de14c8210e5c59c3fb647a82"} Sep 16 13:08:57.503847 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.503830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m2lwv" event={"ID":"d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece","Type":"ContainerStarted","Data":"131eb71916ef2674a82b08a992b61be6691ac6eac5866893a4b74333a1fd7b3a"} Sep 16 13:08:57.520189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.520130 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lv88k" podStartSLOduration=3.584469333 podStartE2EDuration="22.52011228s" totalImagesPullingTime="18.935642947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:57.519906501 +0000 UTC m=+22.639633076" watchObservedRunningTime="2026-09-16 13:08:57.52011228 +0000 UTC m=+22.639838830" Sep 16 13:08:57.551051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.550994 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4ch6p" podStartSLOduration=3.663332291 podStartE2EDuration="22.550973807s" totalImagesPullingTime="18.887641516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:57.550953174 +0000 UTC m=+22.670679715" watchObservedRunningTime="2026-09-16 13:08:57.550973807 +0000 UTC m=+22.670700357" Sep 16 13:08:57.567263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.567209 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hpj8d" podStartSLOduration=3.658042045 podStartE2EDuration="22.567188981s" totalImagesPullingTime="18.909146936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:57.566570808 +0000 UTC m=+22.686297367" watchObservedRunningTime="2026-09-16 13:08:57.567188981 +0000 UTC m=+22.686915531" Sep 16 13:08:57.580596 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.580555 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m2lwv" podStartSLOduration=3.639797668 podStartE2EDuration="22.580536618s" totalImagesPullingTime="18.94073895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:57.580372112 +0000 UTC m=+22.700098661" watchObservedRunningTime="2026-09-16 13:08:57.580536618 +0000 UTC m=+22.700263168" Sep 16 13:08:57.684724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.684635 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:08:57.741368 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.741341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:08:57.842610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:57.842590 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 13:08:58.150696 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.150610 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:08:58.250452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.250420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:08:58.357085 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.356973 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:08:57.842607302Z","UUID":"73e83ae7-a20c-4665-8bf1-8785d26bcfe8","Handler":null,"Name":"","Endpoint":""} Sep 16 13:08:58.358838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.358802 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 13:08:58.358838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.358841 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 13:08:58.380398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.380373 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:08:58.380542 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.380372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:08:58.380542 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:58.380495 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:08:58.380682 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:58.380620 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:08:58.508880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.508800 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" event={"ID":"af97f616135893b3ca8cfcca10b953ab","Type":"ContainerStarted","Data":"9d98274e3d74ab035f54a1857697a9d5799fb4bdecd798ad9e372f82f3952b80"} Sep 16 13:08:58.511062 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.511037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" event={"ID":"57862b11-5908-4bbd-bad9-164a84fce95c","Type":"ContainerStarted","Data":"2630f781ae70f632998d115ae4af1f10bc18c9710b58f824f8126e212b650890"} Sep 16 13:08:58.525002 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:58.524899 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-73.ec2.internal" podStartSLOduration=22.524881349 podStartE2EDuration="22.524881349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:58.524268326 +0000 UTC m=+23.643994876" watchObservedRunningTime="2026-09-16 13:08:58.524881349 +0000 UTC m=+23.644607900" Sep 16 13:08:59.380647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.380461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:08:59.380798 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:08:59.380726 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:08:59.383411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.383388 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:59.383524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.383456 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:08:59.516852 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.516824 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:08:59.517272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.517246 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"b776f573b75216b56196f9718c79a409ae4d155d2c85518d88e3363a4c73287f"} Sep 16 13:08:59.519587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.519560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" event={"ID":"57862b11-5908-4bbd-bad9-164a84fce95c","Type":"ContainerStarted","Data":"ae1a961ba8cd3d769267bf090fc14defb29cb550af9fe3497298915de528448d"} Sep 16 13:08:59.538444 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:08:59.538394 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hj874" podStartSLOduration=3.560644044 podStartE2EDuration="24.538381567s" totalImagesPullingTime="20.977737523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:59.538159221 +0000 UTC m=+24.657885792" watchObservedRunningTime="2026-09-16 13:08:59.538381567 +0000 UTC m=+24.658108116" Sep 16 13:09:00.380990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:00.380963 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:00.380990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:00.380996 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:00.381246 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:00.381076 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:09:00.381246 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:00.381225 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:09:01.010132 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:01.010100 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:09:01.010608 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:01.010221 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 13:09:01.010805 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:01.010789 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hpj8d" Sep 16 13:09:01.068620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:01.068580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:01.380681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:01.380604 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:01.380842 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:01.380740 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:09:02.380593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.380568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:02.380924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.380568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:02.380924 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:02.380699 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:09:02.380924 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:02.380733 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:09:02.530114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.530085 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:09:02.530501 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.530466 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"3157d0ad2b9c5ea921642b72969bd995c58ef9e09e5bbeb697933a05511bf629"} Sep 16 13:09:02.530706 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.530686 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:09:02.530810 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.530774 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:09:02.530881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.530867 2581 scope.go:117] "RemoveContainer" containerID="2f3e82a27d13a75f41da402f2ffcf49c68cc849c10bfe23f0913d1c3a8640e41" Sep 16 13:09:02.535492 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.535463 2581 generic.go:356] "Generic (PLEG): container finished" podID="75bd51a9-a796-46f3-ae52-fb1b49353736" containerID="940ed7420975598cbe6f6ad5c8beae4103ce1eb1828b5fdcc86c196816bc8ac8" exitCode=0 Sep 16 13:09:02.535605 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.535500 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerDied","Data":"940ed7420975598cbe6f6ad5c8beae4103ce1eb1828b5fdcc86c196816bc8ac8"} Sep 16 13:09:02.547571 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:02.547427 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:09:03.381217 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.381187 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:03.381755 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:03.381336 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:09:03.544460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.544434 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:09:03.544835 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.544808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" event={"ID":"0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81","Type":"ContainerStarted","Data":"f877bcd939ef00b1fcf43fb69e6a65bd152fcfb6d845edb8688872969723f1b4"} Sep 16 13:09:03.545056 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.545038 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:09:03.558851 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.558828 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:09:03.574429 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.573262 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" podStartSLOduration=9.647343166 podStartE2EDuration="28.573249095s" totalImagesPullingTime="18.925905929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:03.572922384 +0000 UTC m=+28.692648958" watchObservedRunningTime="2026-09-16 13:09:03.573249095 +0000 UTC m=+28.692975644" Sep 16 13:09:03.980275 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:03.980242 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:04.047273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.047194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:04.094613 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.094586 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zvh5v"] Sep 16 13:09:04.094756 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.094741 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:04.094875 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:04.094854 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:09:04.097614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.097589 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4nw4"] Sep 16 13:09:04.097743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.097694 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:04.097802 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:04.097783 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:09:04.098119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.098094 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zs895"] Sep 16 13:09:04.098220 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.098188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:04.098314 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:04.098267 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:09:04.549161 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.549129 2581 generic.go:356] "Generic (PLEG): container finished" podID="75bd51a9-a796-46f3-ae52-fb1b49353736" containerID="75f51bb657f8afab653708caba7b3290efc34b07db8c94b5a7219f914c069fd0" exitCode=0 Sep 16 13:09:04.549930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:04.549208 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerDied","Data":"75f51bb657f8afab653708caba7b3290efc34b07db8c94b5a7219f914c069fd0"} Sep 16 13:09:05.380983 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:05.380955 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:05.381136 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:05.381066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:05.381136 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:05.381099 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:05.381251 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:05.381059 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:09:05.381251 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:05.381159 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:09:05.381373 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:05.381257 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:09:05.554469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:05.554148 2581 generic.go:356] "Generic (PLEG): container finished" podID="75bd51a9-a796-46f3-ae52-fb1b49353736" containerID="be19e082c9161657cd88dc1ecf51b92085a1474c289bdbec91b5c2602b1a40bc" exitCode=0 Sep 16 13:09:05.554469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:05.554232 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerDied","Data":"be19e082c9161657cd88dc1ecf51b92085a1474c289bdbec91b5c2602b1a40bc"} Sep 16 13:09:07.380603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:07.380564 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:07.381089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:07.380574 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:07.381089 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:07.380693 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zvh5v" podUID="b6dc3916-9abe-4021-a871-0deca7e008f1" Sep 16 13:09:07.381089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:07.380575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:07.381089 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:07.380751 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zs895" podUID="906cc307-f99a-4be6-a72b-38024d094c15" Sep 16 13:09:07.381089 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:07.380817 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4nw4" podUID="bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8" Sep 16 13:09:08.051102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.051029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:08.051239 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.051132 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:09:08.051239 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.051177 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret podName:b6dc3916-9abe-4021-a871-0deca7e008f1 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:40.051164743 +0000 UTC m=+65.170891270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret") pod "global-pull-secret-syncer-zvh5v" (UID: "b6dc3916-9abe-4021-a871-0deca7e008f1") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:09:08.115184 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.115148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:08.151981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.151955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:08.152104 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.152091 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:09:08.152159 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.152150 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs podName:906cc307-f99a-4be6-a72b-38024d094c15 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:40.152133782 +0000 UTC m=+65.271860309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs") pod "network-metrics-daemon-zs895" (UID: "906cc307-f99a-4be6-a72b-38024d094c15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:09:08.219188 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.219158 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:08.253345 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.253317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:08.253465 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.253449 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:09:08.253504 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.253468 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:09:08.253504 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.253481 2581 projected.go:196] Error preparing data for projected volume kube-api-access-9m49c for pod openshift-network-diagnostics/network-check-target-f4nw4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:09:08.253637 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:08.253536 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c podName:bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:40.25351999 +0000 UTC m=+65.373246521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m49c" (UniqueName: "kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c") pod "network-check-target-f4nw4" (UID: "bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:09:08.427170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.427086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:08.490149 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:08.490117 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:09.227026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.227001 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-73.ec2.internal" event="NodeReady" Sep 16 13:09:09.227216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.227125 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 13:09:09.261240 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.261210 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-754f9b4486-mw6bn"] Sep 16 13:09:09.293174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.292916 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lmsv9"] Sep 16 13:09:09.293174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.293019 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.295171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.295150 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 13:09:09.295450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.295428 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 13:09:09.295560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.295524 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lhrr5\"" Sep 16 13:09:09.295759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.295740 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 13:09:09.300610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.300589 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 13:09:09.308123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.308094 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-24dv4"] Sep 16 13:09:09.308240 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.308219 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.310321 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.310301 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 13:09:09.310420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.310400 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftxst\"" Sep 16 13:09:09.310478 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.310434 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 13:09:09.326553 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.326520 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.326655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.326522 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-754f9b4486-mw6bn"] Sep 16 13:09:09.326718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.326685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmsv9"] Sep 16 13:09:09.326773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.326724 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-24dv4"] Sep 16 13:09:09.326773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.326747 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzkfk"] Sep 16 13:09:09.339272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.339249 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dbtv4\"" Sep 16 13:09:09.339582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.339548 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 13:09:09.339814 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.339799 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 13:09:09.342481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.340687 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 13:09:09.354653 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.354633 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzkfk"] Sep 16 13:09:09.354867 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.354850 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.357063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.357045 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 13:09:09.357063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.357056 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dlvxs\"" Sep 16 13:09:09.357208 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.357060 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 13:09:09.357208 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.357098 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 13:09:09.357454 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.357439 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 13:09:09.381306 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.381269 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:09.381433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.381290 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:09.381543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.381525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:09.383656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.383637 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:09:09.383757 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.383744 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:09:09.383853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.383778 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:09:09.383853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.383796 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:09:09.384079 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.384064 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sfm4x\"" Sep 16 13:09:09.384161 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.384121 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5rcg\"" Sep 16 13:09:09.461190 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdd31a-1830-4d96-8a94-f3083f6d9b85-config-volume\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.461190 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6nx\" (UniqueName: \"kubernetes.io/projected/90fdd31a-1830-4d96-8a94-f3083f6d9b85-kube-api-access-br6nx\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461195 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lltr\" (UniqueName: \"kubernetes.io/projected/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-api-access-5lltr\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461347 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-certificates\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a8564d5-0e0d-4262-8b90-3af68cb8574e-data-volume\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461412 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67q98\" (UniqueName: \"kubernetes.io/projected/cd78deae-72ad-4d9c-a31d-e423437d8169-kube-api-access-67q98\") pod \"ingress-canary-24dv4\" (UID: \"cd78deae-72ad-4d9c-a31d-e423437d8169\") " pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9ce0f05-fb33-413b-be8d-11219809b2b9-ca-trust-extracted\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461524 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-image-registry-private-configuration\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd78deae-72ad-4d9c-a31d-e423437d8169-cert\") pod \"ingress-canary-24dv4\" (UID: \"cd78deae-72ad-4d9c-a31d-e423437d8169\") " pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.461641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-trusted-ca\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.462043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-bound-sa-token\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.462043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90fdd31a-1830-4d96-8a94-f3083f6d9b85-tmp-dir\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.462043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-installation-pull-secrets\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.462043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlq4\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-kube-api-access-rtlq4\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.462043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a8564d5-0e0d-4262-8b90-3af68cb8574e-crio-socket\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.462043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.461825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.561143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.561116 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m2lwv_d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece/dns-node-resolver/0.log" Sep 16 13:09:09.562298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdd31a-1830-4d96-8a94-f3083f6d9b85-config-volume\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.562419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562306 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6nx\" (UniqueName: \"kubernetes.io/projected/90fdd31a-1830-4d96-8a94-f3083f6d9b85-kube-api-access-br6nx\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.562419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lltr\" (UniqueName: \"kubernetes.io/projected/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-api-access-5lltr\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.562419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-certificates\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a8564d5-0e0d-4262-8b90-3af68cb8574e-data-volume\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.562593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562438 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67q98\" (UniqueName: \"kubernetes.io/projected/cd78deae-72ad-4d9c-a31d-e423437d8169-kube-api-access-67q98\") pod \"ingress-canary-24dv4\" (UID: \"cd78deae-72ad-4d9c-a31d-e423437d8169\") " pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.562593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9ce0f05-fb33-413b-be8d-11219809b2b9-ca-trust-extracted\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562509 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.562593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-image-registry-private-configuration\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.562737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd78deae-72ad-4d9c-a31d-e423437d8169-cert\") pod \"ingress-canary-24dv4\" (UID: \"cd78deae-72ad-4d9c-a31d-e423437d8169\") " pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.562737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-trusted-ca\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-bound-sa-token\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90fdd31a-1830-4d96-8a94-f3083f6d9b85-tmp-dir\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.562882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-installation-pull-secrets\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlq4\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-kube-api-access-rtlq4\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.562882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a8564d5-0e0d-4262-8b90-3af68cb8574e-crio-socket\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.562882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.563006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562886 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90fdd31a-1830-4d96-8a94-f3083f6d9b85-config-volume\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.563006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562917 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9ce0f05-fb33-413b-be8d-11219809b2b9-ca-trust-extracted\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.563006 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.562974 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:09:09.563006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.562980 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-certificates\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.563121 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.563044 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:10.063027519 +0000 UTC m=+35.182754062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : configmap "kube-rbac-proxy" not found Sep 16 13:09:09.563177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.563159 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a8564d5-0e0d-4262-8b90-3af68cb8574e-data-volume\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.563254 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.563241 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:09:09.563328 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.563317 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:10.06330333 +0000 UTC m=+35.183029856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : secret "insights-runtime-extractor-tls" not found Sep 16 13:09:09.563383 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.563366 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:09:09.563422 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.563386 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-754f9b4486-mw6bn: secret "image-registry-tls" not found Sep 16 13:09:09.563458 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.563431 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls podName:c9ce0f05-fb33-413b-be8d-11219809b2b9 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:10.0634158 +0000 UTC m=+35.183142330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls") pod "image-registry-754f9b4486-mw6bn" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9") : secret "image-registry-tls" not found Sep 16 13:09:09.563561 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.563546 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90fdd31a-1830-4d96-8a94-f3083f6d9b85-tmp-dir\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.563847 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.563829 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a8564d5-0e0d-4262-8b90-3af68cb8574e-crio-socket\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.564269 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.564252 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:09:09.564380 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:09.564325 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls podName:90fdd31a-1830-4d96-8a94-f3083f6d9b85 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:10.064311082 +0000 UTC m=+35.184037609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls") pod "dns-default-lmsv9" (UID: "90fdd31a-1830-4d96-8a94-f3083f6d9b85") : secret "dns-default-metrics-tls" not found Sep 16 13:09:09.564849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.564826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-trusted-ca\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.568133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.568107 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-image-registry-private-configuration\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.568221 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.568108 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-installation-pull-secrets\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.568221 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.568139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd78deae-72ad-4d9c-a31d-e423437d8169-cert\") pod \"ingress-canary-24dv4\" (UID: \"cd78deae-72ad-4d9c-a31d-e423437d8169\") " pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.572132 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.572107 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6nx\" (UniqueName: \"kubernetes.io/projected/90fdd31a-1830-4d96-8a94-f3083f6d9b85-kube-api-access-br6nx\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:09.572348 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.572325 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-bound-sa-token\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.572464 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.572444 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lltr\" (UniqueName: \"kubernetes.io/projected/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-api-access-5lltr\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:09.572923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.572903 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67q98\" (UniqueName: \"kubernetes.io/projected/cd78deae-72ad-4d9c-a31d-e423437d8169-kube-api-access-67q98\") pod \"ingress-canary-24dv4\" (UID: \"cd78deae-72ad-4d9c-a31d-e423437d8169\") " pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.573801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.573783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlq4\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-kube-api-access-rtlq4\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:09.649538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.649491 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-24dv4" Sep 16 13:09:09.754559 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.754482 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn"] Sep 16 13:09:09.772507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.772482 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn"] Sep 16 13:09:09.772637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.772593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.775018 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.774995 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9fwlk\"" Sep 16 13:09:09.775202 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.775040 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 13:09:09.775202 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.775065 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 13:09:09.775662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.775458 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:09:09.775662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.775486 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 13:09:09.864993 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.864954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-serving-cert\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.865154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.865092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-config\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.865239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.865219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5g9\" (UniqueName: \"kubernetes.io/projected/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-kube-api-access-9s5g9\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.966356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.966308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-config\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.966520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.966439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5g9\" (UniqueName: \"kubernetes.io/projected/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-kube-api-access-9s5g9\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.966520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.966472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-serving-cert\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.966926 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.966904 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-config\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.969128 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.969104 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-serving-cert\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.980691 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.980668 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5g9\" (UniqueName: \"kubernetes.io/projected/5072e389-8c92-4961-b1f8-8cdc1cc4fa33-kube-api-access-9s5g9\") pod \"service-ca-operator-cf7c776b7-jk6kn\" (UID: \"5072e389-8c92-4961-b1f8-8cdc1cc4fa33\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:09.995942 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:09.995924 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pphw5"] Sep 16 13:09:10.020691 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.020627 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pphw5"] Sep 16 13:09:10.020789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.020722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" Sep 16 13:09:10.023886 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.023868 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wsz5x\"" Sep 16 13:09:10.067364 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.067341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:10.067507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.067373 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:10.067507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.067397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:10.067507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.067455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:10.067507 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067499 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067510 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067536 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067553 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:11.067536443 +0000 UTC m=+36.187262986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : secret "insights-runtime-extractor-tls" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067580 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067592 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls podName:90fdd31a-1830-4d96-8a94-f3083f6d9b85 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:11.067581933 +0000 UTC m=+36.187308461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls") pod "dns-default-lmsv9" (UID: "90fdd31a-1830-4d96-8a94-f3083f6d9b85") : secret "dns-default-metrics-tls" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067595 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-754f9b4486-mw6bn: secret "image-registry-tls" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067602 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:11.067597292 +0000 UTC m=+36.187323818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : configmap "kube-rbac-proxy" not found Sep 16 13:09:10.067717 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:10.067637 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls podName:c9ce0f05-fb33-413b-be8d-11219809b2b9 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:11.067617191 +0000 UTC m=+36.187343718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls") pod "image-registry-754f9b4486-mw6bn" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9") : secret "image-registry-tls" not found Sep 16 13:09:10.083130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.083106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" Sep 16 13:09:10.168719 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.168687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2rx\" (UniqueName: \"kubernetes.io/projected/aafed7fb-3583-4004-bd8d-fb624c006519-kube-api-access-2z2rx\") pod \"network-check-source-6f8b768455-pphw5\" (UID: \"aafed7fb-3583-4004-bd8d-fb624c006519\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" Sep 16 13:09:10.269409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.269371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2rx\" (UniqueName: \"kubernetes.io/projected/aafed7fb-3583-4004-bd8d-fb624c006519-kube-api-access-2z2rx\") pod \"network-check-source-6f8b768455-pphw5\" (UID: \"aafed7fb-3583-4004-bd8d-fb624c006519\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" Sep 16 13:09:10.278850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.278769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2rx\" (UniqueName: \"kubernetes.io/projected/aafed7fb-3583-4004-bd8d-fb624c006519-kube-api-access-2z2rx\") pod \"network-check-source-6f8b768455-pphw5\" (UID: \"aafed7fb-3583-4004-bd8d-fb624c006519\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" Sep 16 13:09:10.330828 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.330800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" Sep 16 13:09:10.361009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.360989 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lv88k_00be8b34-1abf-4801-b955-ba483d5f6c34/node-ca/0.log" Sep 16 13:09:10.820466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.820030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn"] Sep 16 13:09:10.820466 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:10.820086 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5072e389_8c92_4961_b1f8_8cdc1cc4fa33.slice/crio-53b676ab534e80696e35950827f4d7265760754799075e29b4dd6042e685a85e WatchSource:0}: Error finding container 53b676ab534e80696e35950827f4d7265760754799075e29b4dd6042e685a85e: Status 404 returned error can't find the container with id 53b676ab534e80696e35950827f4d7265760754799075e29b4dd6042e685a85e Sep 16 13:09:10.822388 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.822309 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-24dv4"] Sep 16 13:09:10.823318 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:10.823297 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pphw5"] Sep 16 13:09:10.824474 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:10.824449 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafed7fb_3583_4004_bd8d_fb624c006519.slice/crio-576d3cfda40bf1ff539b24f1111987b23bea1dcff41d11a74eead032434dcf07 WatchSource:0}: Error finding container 576d3cfda40bf1ff539b24f1111987b23bea1dcff41d11a74eead032434dcf07: Status 404 returned error can't find the container with id 576d3cfda40bf1ff539b24f1111987b23bea1dcff41d11a74eead032434dcf07 Sep 16 13:09:10.825549 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:10.825525 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd78deae_72ad_4d9c_a31d_e423437d8169.slice/crio-1fd204aea9f134ee2243f93fa157fbd3eef36f18d5ac86bbe02f03c8995c3cc4 WatchSource:0}: Error finding container 1fd204aea9f134ee2243f93fa157fbd3eef36f18d5ac86bbe02f03c8995c3cc4: Status 404 returned error can't find the container with id 1fd204aea9f134ee2243f93fa157fbd3eef36f18d5ac86bbe02f03c8995c3cc4 Sep 16 13:09:11.031633 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.031598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:11.053178 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.053120 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc"] Sep 16 13:09:11.070921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.070900 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc"] Sep 16 13:09:11.071030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.070999 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" Sep 16 13:09:11.073915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.073892 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 13:09:11.073915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.073908 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 13:09:11.074099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.073999 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-htfnd\"" Sep 16 13:09:11.075986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.075968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:11.076087 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.076006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:11.076087 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.076067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:11.076087 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076074 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:09:11.076239 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076144 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:09:11.076239 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076155 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls podName:90fdd31a-1830-4d96-8a94-f3083f6d9b85 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:13.076135012 +0000 UTC m=+38.195861567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls") pod "dns-default-lmsv9" (UID: "90fdd31a-1830-4d96-8a94-f3083f6d9b85") : secret "dns-default-metrics-tls" not found Sep 16 13:09:11.076239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.076143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:11.076239 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076158 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-754f9b4486-mw6bn: secret "image-registry-tls" not found Sep 16 13:09:11.076239 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076238 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls podName:c9ce0f05-fb33-413b-be8d-11219809b2b9 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:13.076226834 +0000 UTC m=+38.195953375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls") pod "image-registry-754f9b4486-mw6bn" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9") : secret "image-registry-tls" not found Sep 16 13:09:11.076474 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076251 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:09:11.076474 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076338 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:13.076318056 +0000 UTC m=+38.196044600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : configmap "kube-rbac-proxy" not found Sep 16 13:09:11.076474 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076355 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:09:11.076474 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:11.076402 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:13.076387846 +0000 UTC m=+38.196114384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : secret "insights-runtime-extractor-tls" not found Sep 16 13:09:11.127635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.127613 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:11.176773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.176750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm6rt\" (UniqueName: \"kubernetes.io/projected/41ad6f3d-f396-4a4c-8754-b2126090c34b-kube-api-access-jm6rt\") pod \"migrator-8b4d697c5-fwkkc\" (UID: \"41ad6f3d-f396-4a4c-8754-b2126090c34b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" Sep 16 13:09:11.224055 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.224031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:11.277260 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.277236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6rt\" (UniqueName: \"kubernetes.io/projected/41ad6f3d-f396-4a4c-8754-b2126090c34b-kube-api-access-jm6rt\") pod \"migrator-8b4d697c5-fwkkc\" (UID: \"41ad6f3d-f396-4a4c-8754-b2126090c34b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" Sep 16 13:09:11.296761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.296739 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6rt\" (UniqueName: \"kubernetes.io/projected/41ad6f3d-f396-4a4c-8754-b2126090c34b-kube-api-access-jm6rt\") pod \"migrator-8b4d697c5-fwkkc\" (UID: \"41ad6f3d-f396-4a4c-8754-b2126090c34b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" Sep 16 13:09:11.379412 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.379339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" Sep 16 13:09:11.397141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.397106 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:11.495460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.495424 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:11.573843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.573806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-24dv4" event={"ID":"cd78deae-72ad-4d9c-a31d-e423437d8169","Type":"ContainerStarted","Data":"1fd204aea9f134ee2243f93fa157fbd3eef36f18d5ac86bbe02f03c8995c3cc4"} Sep 16 13:09:11.578210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.578176 2581 generic.go:356] "Generic (PLEG): container finished" podID="75bd51a9-a796-46f3-ae52-fb1b49353736" containerID="be26a553cab3f681ee5bd034b996b720e71c6a8fe9522b8ff0bfd91d5f63e085" exitCode=0 Sep 16 13:09:11.578375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.578221 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerDied","Data":"be26a553cab3f681ee5bd034b996b720e71c6a8fe9522b8ff0bfd91d5f63e085"} Sep 16 13:09:11.580053 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.579840 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" event={"ID":"5072e389-8c92-4961-b1f8-8cdc1cc4fa33","Type":"ContainerStarted","Data":"53b676ab534e80696e35950827f4d7265760754799075e29b4dd6042e685a85e"} Sep 16 13:09:11.581313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.581271 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" event={"ID":"aafed7fb-3583-4004-bd8d-fb624c006519","Type":"ContainerStarted","Data":"576d3cfda40bf1ff539b24f1111987b23bea1dcff41d11a74eead032434dcf07"} Sep 16 13:09:11.595314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.595268 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:11.666239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:11.666147 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc"] Sep 16 13:09:11.672315 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:11.672263 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ad6f3d_f396_4a4c_8754_b2126090c34b.slice/crio-519df7a5764ae62bd44760f28fa0259f3b6e0a90dbedc1e6d5eca584cc6e766f WatchSource:0}: Error finding container 519df7a5764ae62bd44760f28fa0259f3b6e0a90dbedc1e6d5eca584cc6e766f: Status 404 returned error can't find the container with id 519df7a5764ae62bd44760f28fa0259f3b6e0a90dbedc1e6d5eca584cc6e766f Sep 16 13:09:12.319060 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:12.318962 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:12.418319 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:12.418254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:12.476298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:12.475968 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:12.591604 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:12.589384 2581 generic.go:356] "Generic (PLEG): container finished" podID="75bd51a9-a796-46f3-ae52-fb1b49353736" containerID="5197b09af7d0fd1e0ae26f689451103a9ae5f5fb1b793845f897f95afc0f1a07" exitCode=0 Sep 16 13:09:12.591604 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:12.589505 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerDied","Data":"5197b09af7d0fd1e0ae26f689451103a9ae5f5fb1b793845f897f95afc0f1a07"} Sep 16 13:09:12.592908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:12.592877 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" event={"ID":"41ad6f3d-f396-4a4c-8754-b2126090c34b","Type":"ContainerStarted","Data":"519df7a5764ae62bd44760f28fa0259f3b6e0a90dbedc1e6d5eca584cc6e766f"} Sep 16 13:09:13.096489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.096204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:13.096686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.096514 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:13.096686 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096428 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:09:13.096686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.096601 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:13.096686 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096628 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:17.09660586 +0000 UTC m=+42.216332401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : secret "insights-runtime-extractor-tls" not found Sep 16 13:09:13.096686 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096667 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:09:13.096686 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096686 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:09:13.097053 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096699 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-754f9b4486-mw6bn: secret "image-registry-tls" not found Sep 16 13:09:13.097053 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.096711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:13.097053 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096723 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls podName:90fdd31a-1830-4d96-8a94-f3083f6d9b85 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:17.096706024 +0000 UTC m=+42.216432560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls") pod "dns-default-lmsv9" (UID: "90fdd31a-1830-4d96-8a94-f3083f6d9b85") : secret "dns-default-metrics-tls" not found Sep 16 13:09:13.097053 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096748 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls podName:c9ce0f05-fb33-413b-be8d-11219809b2b9 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:17.09673824 +0000 UTC m=+42.216464770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls") pod "image-registry-754f9b4486-mw6bn" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9") : secret "image-registry-tls" not found Sep 16 13:09:13.097053 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096761 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:09:13.097053 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:13.096799 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:17.096787158 +0000 UTC m=+42.216513701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : configmap "kube-rbac-proxy" not found Sep 16 13:09:13.599179 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.599138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" event={"ID":"75bd51a9-a796-46f3-ae52-fb1b49353736","Type":"ContainerStarted","Data":"9cbaa15aff298183470e63ffbc1309cc189336ee511ee8bb6c53d347f07ca883"} Sep 16 13:09:13.620610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.620568 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bmbbz" podStartSLOduration=7.936413643 podStartE2EDuration="38.620547462s" totalImagesPullingTime="30.684133819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:13.619156374 +0000 UTC m=+38.738882981" watchObservedRunningTime="2026-09-16 13:09:13.620547462 +0000 UTC m=+38.740274013" Sep 16 13:09:13.740394 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.740354 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:13.842114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:13.842084 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:14.401573 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:14.401526 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:14.502444 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:14.502390 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:14.742441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:14.742372 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:14.802215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:14.802174 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:15.758733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:15.758704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:15.816002 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:15.815968 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:16.608369 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.608327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-24dv4" event={"ID":"cd78deae-72ad-4d9c-a31d-e423437d8169","Type":"ContainerStarted","Data":"5f414e6e084cb575d50ef0ba4117defea1e7301bdfae945139579680907a735d"} Sep 16 13:09:16.609947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.609921 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" event={"ID":"5072e389-8c92-4961-b1f8-8cdc1cc4fa33","Type":"ContainerStarted","Data":"b0a230ec0d2b704209fb077650947fd8f6538f135f9dfb5ac9c8564d8fe7547a"} Sep 16 13:09:16.611827 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.611804 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" event={"ID":"41ad6f3d-f396-4a4c-8754-b2126090c34b","Type":"ContainerStarted","Data":"685a709c51735f97709110d33ea833cdbf0ee19ff54211eefe52dffd72d0becb"} Sep 16 13:09:16.611938 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.611858 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" event={"ID":"41ad6f3d-f396-4a4c-8754-b2126090c34b","Type":"ContainerStarted","Data":"d10c86b15dde2485d73ff3b2c57f87fd7c7040d8f90c6417236de4913fc3faa6"} Sep 16 13:09:16.613270 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.613251 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" event={"ID":"aafed7fb-3583-4004-bd8d-fb624c006519","Type":"ContainerStarted","Data":"d5a02b4d9c1926f523e9e6d8491540dda96edae9cbf9a60c5e137e3bd7aef0b2"} Sep 16 13:09:16.624165 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.624127 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-24dv4" podStartSLOduration=2.281582057 podStartE2EDuration="7.624115745s" totalImagesPullingTime="5.342533688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:16.622711407 +0000 UTC m=+41.742437957" watchObservedRunningTime="2026-09-16 13:09:16.624115745 +0000 UTC m=+41.743842294" Sep 16 13:09:16.637770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.637732 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pphw5" podStartSLOduration=2.285490164 podStartE2EDuration="7.637721496s" totalImagesPullingTime="5.352231332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:16.636694514 +0000 UTC m=+41.756421073" watchObservedRunningTime="2026-09-16 13:09:16.637721496 +0000 UTC m=+41.757448036" Sep 16 13:09:16.650933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.650878 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fwkkc" podStartSLOduration=1.156469447 podStartE2EDuration="5.650861397s" totalImagesPullingTime="4.49439195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:16.650576852 +0000 UTC m=+41.770303423" watchObservedRunningTime="2026-09-16 13:09:16.650861397 +0000 UTC m=+41.770587949" Sep 16 13:09:16.667486 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.667450 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" podStartSLOduration=2.3200024949999998 podStartE2EDuration="7.667437438s" totalImagesPullingTime="5.347434943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:16.666581281 +0000 UTC m=+41.786307854" watchObservedRunningTime="2026-09-16 13:09:16.667437438 +0000 UTC m=+41.787163987" Sep 16 13:09:16.755432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.755344 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:16.872590 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:16.872548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:17.131924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:17.131833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:17.131924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:17.131883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:17.131946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.131976 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:17.132015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132034 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls podName:90fdd31a-1830-4d96-8a94-f3083f6d9b85 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:25.132020432 +0000 UTC m=+50.251746958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls") pod "dns-default-lmsv9" (UID: "90fdd31a-1830-4d96-8a94-f3083f6d9b85") : secret "dns-default-metrics-tls" not found Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132061 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132091 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-754f9b4486-mw6bn: secret "image-registry-tls" not found Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132097 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132105 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:09:17.132151 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132142 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:25.132126546 +0000 UTC m=+50.251853092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : secret "insights-runtime-extractor-tls" not found Sep 16 13:09:17.132453 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132161 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls podName:c9ce0f05-fb33-413b-be8d-11219809b2b9 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:25.132152079 +0000 UTC m=+50.251878606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls") pod "image-registry-754f9b4486-mw6bn" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9") : secret "image-registry-tls" not found Sep 16 13:09:17.132453 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:17.132176 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:25.132167747 +0000 UTC m=+50.251894279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : configmap "kube-rbac-proxy" not found Sep 16 13:09:18.139610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:18.139575 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:18.239295 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:18.239238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:18.841419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:18.841346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:18.941644 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:18.941608 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:20.442386 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.442358 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-cgqh2"] Sep 16 13:09:20.465699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.465676 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-cgqh2"] Sep 16 13:09:20.465848 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.465761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.468452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.468428 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 16 13:09:20.469099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.469080 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 16 13:09:20.469201 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.469149 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 16 13:09:20.469201 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.469087 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5k65c\"" Sep 16 13:09:20.469201 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.469184 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 16 13:09:20.499765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.499739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:20.554110 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.554086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:20.557875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.557858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj74p\" (UniqueName: \"kubernetes.io/projected/1107970f-cbeb-48ff-8081-fa6e81f957a9-kube-api-access-lj74p\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.557948 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.557927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1107970f-cbeb-48ff-8081-fa6e81f957a9-signing-cabundle\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.558002 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.557969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1107970f-cbeb-48ff-8081-fa6e81f957a9-signing-key\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.658852 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.658829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1107970f-cbeb-48ff-8081-fa6e81f957a9-signing-key\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.658989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.658898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj74p\" (UniqueName: \"kubernetes.io/projected/1107970f-cbeb-48ff-8081-fa6e81f957a9-kube-api-access-lj74p\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.658989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.658977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1107970f-cbeb-48ff-8081-fa6e81f957a9-signing-cabundle\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.659649 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.659623 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1107970f-cbeb-48ff-8081-fa6e81f957a9-signing-cabundle\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.662027 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.662008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1107970f-cbeb-48ff-8081-fa6e81f957a9-signing-key\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.667061 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.667039 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj74p\" (UniqueName: \"kubernetes.io/projected/1107970f-cbeb-48ff-8081-fa6e81f957a9-kube-api-access-lj74p\") pod \"service-ca-568869fdc5-cgqh2\" (UID: \"1107970f-cbeb-48ff-8081-fa6e81f957a9\") " pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:20.774757 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:20.774734 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" Sep 16 13:09:21.076634 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:21.076553 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-cgqh2"] Sep 16 13:09:21.080271 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:21.080249 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1107970f_cbeb_48ff_8081_fa6e81f957a9.slice/crio-7c8b6cd529a736a9c2fb123f0af99606eb15169a5432f59a148e746d14bb63a7 WatchSource:0}: Error finding container 7c8b6cd529a736a9c2fb123f0af99606eb15169a5432f59a148e746d14bb63a7: Status 404 returned error can't find the container with id 7c8b6cd529a736a9c2fb123f0af99606eb15169a5432f59a148e746d14bb63a7 Sep 16 13:09:21.519741 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:21.519701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:21.622355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:21.622325 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:21.633109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:21.633080 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" event={"ID":"1107970f-cbeb-48ff-8081-fa6e81f957a9","Type":"ContainerStarted","Data":"cdb7106ba7e64df9b2fae24be0977ec74af9cc923a18a10e30ccdad3c10d7785"} Sep 16 13:09:21.633254 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:21.633128 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" event={"ID":"1107970f-cbeb-48ff-8081-fa6e81f957a9","Type":"ContainerStarted","Data":"7c8b6cd529a736a9c2fb123f0af99606eb15169a5432f59a148e746d14bb63a7"} Sep 16 13:09:22.744531 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:22.744496 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:22.978114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:22.978077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:23.076961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:23.076880 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:23.358964 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:23.358874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:23.459462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:23.459425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:23.594439 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:23.594402 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:23.681485 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:23.681397 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:23.836013 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:23.835978 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:25.055276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.055234 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:25.192579 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.192537 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:25.192579 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.192579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:25.192763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.192612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:25.192763 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192669 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:09:25.192763 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192731 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls podName:1a8564d5-0e0d-4262-8b90-3af68cb8574e nodeName:}" failed. No retries permitted until 2026-09-16 13:09:41.192716575 +0000 UTC m=+66.312443102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzkfk" (UID: "1a8564d5-0e0d-4262-8b90-3af68cb8574e") : secret "insights-runtime-extractor-tls" not found Sep 16 13:09:25.192763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.192671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:25.192929 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192761 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:09:25.192929 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192765 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:09:25.192929 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192826 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-754f9b4486-mw6bn: secret "image-registry-tls" not found Sep 16 13:09:25.192929 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192813 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls podName:90fdd31a-1830-4d96-8a94-f3083f6d9b85 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:41.192800111 +0000 UTC m=+66.312526642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls") pod "dns-default-lmsv9" (UID: "90fdd31a-1830-4d96-8a94-f3083f6d9b85") : secret "dns-default-metrics-tls" not found Sep 16 13:09:25.192929 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:25.192906 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls podName:c9ce0f05-fb33-413b-be8d-11219809b2b9 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:41.192893293 +0000 UTC m=+66.312619832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls") pod "image-registry-754f9b4486-mw6bn" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9") : secret "image-registry-tls" not found Sep 16 13:09:25.193204 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.193187 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a8564d5-0e0d-4262-8b90-3af68cb8574e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:25.256113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.256082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:25.455015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.454963 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:25.644467 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.644412 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:25.719668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.719576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:25.842180 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:25.842147 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:26.458188 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:26.458155 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:26.642481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:26.642444 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:26.837169 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:26.837087 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:27.231264 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:27.231234 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:27.438749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:27.438714 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:27.530521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:27.530434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:27.635632 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:27.635600 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:27.872872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:27.872796 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:27.941172 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:27.941136 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:28.046692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:28.046665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:28.188111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:28.188035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:28.282334 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:28.282300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:28.881883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:28.881850 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:29.043856 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:29.043822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:29.284175 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:29.284141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:29.431238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:29.431208 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:29.641109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:29.641025 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:30.072707 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:30.072674 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:30.632763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:30.632733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:31.040358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:31.040323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:31.436524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:31.436445 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:31.835937 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:31.835904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:32.234173 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:32.234133 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:32.944955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:32.944922 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:33.045531 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:33.045499 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:33.246182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:33.246136 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:33.484098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:33.484071 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:35.564946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:35.564914 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7mng" Sep 16 13:09:35.592072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:35.592024 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-cgqh2" podStartSLOduration=15.592008103 podStartE2EDuration="15.592008103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:21.65721646 +0000 UTC m=+46.776943013" watchObservedRunningTime="2026-09-16 13:09:35.592008103 +0000 UTC m=+60.711734653" Sep 16 13:09:39.079933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:39.079895 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:39.151194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:39.151163 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:39.301993 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:39.301960 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:39.401008 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:39.400929 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:40.118725 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.118694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:40.121444 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.121428 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:09:40.131448 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.131429 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6dc3916-9abe-4021-a871-0deca7e008f1-original-pull-secret\") pod \"global-pull-secret-syncer-zvh5v\" (UID: \"b6dc3916-9abe-4021-a871-0deca7e008f1\") " pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:40.219004 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.218978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:40.221247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.221233 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:09:40.231266 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.231242 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906cc307-f99a-4be6-a72b-38024d094c15-metrics-certs\") pod \"network-metrics-daemon-zs895\" (UID: \"906cc307-f99a-4be6-a72b-38024d094c15\") " pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:40.294477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.294456 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f5rcg\"" Sep 16 13:09:40.298610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.298597 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zvh5v" Sep 16 13:09:40.302723 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.302709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zs895" Sep 16 13:09:40.319792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.319770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:40.322217 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.322199 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m49c\" (UniqueName: \"kubernetes.io/projected/bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8-kube-api-access-9m49c\") pod \"network-check-target-f4nw4\" (UID: \"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8\") " pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:40.574325 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.574297 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zvh5v"] Sep 16 13:09:40.577301 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:40.577259 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6dc3916_9abe_4021_a871_0deca7e008f1.slice/crio-870fb0cd57e4e93a9d469d0fd3d4346f2fb10e7658c4dc46f24f297d733ba27f WatchSource:0}: Error finding container 870fb0cd57e4e93a9d469d0fd3d4346f2fb10e7658c4dc46f24f297d733ba27f: Status 404 returned error can't find the container with id 870fb0cd57e4e93a9d469d0fd3d4346f2fb10e7658c4dc46f24f297d733ba27f Sep 16 13:09:40.577465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.577448 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zs895"] Sep 16 13:09:40.580323 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:40.580305 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906cc307_f99a_4be6_a72b_38024d094c15.slice/crio-4b9d8e8016ff90047901c5ec1737fd79bbb913e4eaa6d9432d28828a11917697 WatchSource:0}: Error finding container 4b9d8e8016ff90047901c5ec1737fd79bbb913e4eaa6d9432d28828a11917697: Status 404 returned error can't find the container with id 4b9d8e8016ff90047901c5ec1737fd79bbb913e4eaa6d9432d28828a11917697 Sep 16 13:09:40.606439 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.606420 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sfm4x\"" Sep 16 13:09:40.614474 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.614460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:40.692347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.692314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zvh5v" event={"ID":"b6dc3916-9abe-4021-a871-0deca7e008f1","Type":"ContainerStarted","Data":"870fb0cd57e4e93a9d469d0fd3d4346f2fb10e7658c4dc46f24f297d733ba27f"} Sep 16 13:09:40.693332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.693308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zs895" event={"ID":"906cc307-f99a-4be6-a72b-38024d094c15","Type":"ContainerStarted","Data":"4b9d8e8016ff90047901c5ec1737fd79bbb913e4eaa6d9432d28828a11917697"} Sep 16 13:09:40.892304 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:40.892223 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4nw4"] Sep 16 13:09:40.895259 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:40.895230 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7d7f61_70e9_4fe7_bbe1_8b766fa403f8.slice/crio-b9e35baee3ebb792f1c5951e46c6d888d5883bfaa404ab675ac1f00a1a5a247a WatchSource:0}: Error finding container b9e35baee3ebb792f1c5951e46c6d888d5883bfaa404ab675ac1f00a1a5a247a: Status 404 returned error can't find the container with id b9e35baee3ebb792f1c5951e46c6d888d5883bfaa404ab675ac1f00a1a5a247a Sep 16 13:09:41.225901 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.225863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:41.226379 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.225911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:41.226379 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.225940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:41.228748 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.228722 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90fdd31a-1830-4d96-8a94-f3083f6d9b85-metrics-tls\") pod \"dns-default-lmsv9\" (UID: \"90fdd31a-1830-4d96-8a94-f3083f6d9b85\") " pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:41.228872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.228785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"image-registry-754f9b4486-mw6bn\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:41.228872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.228801 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a8564d5-0e0d-4262-8b90-3af68cb8574e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzkfk\" (UID: \"1a8564d5-0e0d-4262-8b90-3af68cb8574e\") " pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:41.407598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.407566 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lhrr5\"" Sep 16 13:09:41.415992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.415967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:41.420833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.420804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftxst\"" Sep 16 13:09:41.428807 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.428329 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:41.467460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.467262 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dlvxs\"" Sep 16 13:09:41.477746 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.475101 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzkfk" Sep 16 13:09:41.699441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.699405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4nw4" event={"ID":"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8","Type":"ContainerStarted","Data":"1712f871bb0d97962efe68d20122a59a9994bae369a777e1f7b7b5895476af9c"} Sep 16 13:09:41.699620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.699464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4nw4" event={"ID":"bd7d7f61-70e9-4fe7-bbe1-8b766fa403f8","Type":"ContainerStarted","Data":"b9e35baee3ebb792f1c5951e46c6d888d5883bfaa404ab675ac1f00a1a5a247a"} Sep 16 13:09:41.699620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.699586 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:09:41.707709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.707599 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-754f9b4486-mw6bn"] Sep 16 13:09:41.713247 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:41.713219 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ce0f05_fb33_413b_be8d_11219809b2b9.slice/crio-8ba714b685e0e02ccd0844e2e972ca6119b12e9a9477ac75cd51c83b986e3f32 WatchSource:0}: Error finding container 8ba714b685e0e02ccd0844e2e972ca6119b12e9a9477ac75cd51c83b986e3f32: Status 404 returned error can't find the container with id 8ba714b685e0e02ccd0844e2e972ca6119b12e9a9477ac75cd51c83b986e3f32 Sep 16 13:09:41.721538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.721244 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f4nw4" podStartSLOduration=66.721226868 podStartE2EDuration="1m6.721226868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:41.720853979 +0000 UTC m=+66.840580553" watchObservedRunningTime="2026-09-16 13:09:41.721226868 +0000 UTC m=+66.840953422" Sep 16 13:09:41.739162 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.739129 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmsv9"] Sep 16 13:09:41.764103 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:41.764035 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fdd31a_1830_4d96_8a94_f3083f6d9b85.slice/crio-ce67760d5505aca9452e93d88dd73e36fb38a5617ea9e0d75bd01b4d2f3e8f37 WatchSource:0}: Error finding container ce67760d5505aca9452e93d88dd73e36fb38a5617ea9e0d75bd01b4d2f3e8f37: Status 404 returned error can't find the container with id ce67760d5505aca9452e93d88dd73e36fb38a5617ea9e0d75bd01b4d2f3e8f37 Sep 16 13:09:41.991621 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:41.991545 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzkfk"] Sep 16 13:09:41.995088 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:41.995056 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8564d5_0e0d_4262_8b90_3af68cb8574e.slice/crio-71f81917b17bc6bde7b4010ff36bc84198cd03ba227ce2599cf55e86af9653c7 WatchSource:0}: Error finding container 71f81917b17bc6bde7b4010ff36bc84198cd03ba227ce2599cf55e86af9653c7: Status 404 returned error can't find the container with id 71f81917b17bc6bde7b4010ff36bc84198cd03ba227ce2599cf55e86af9653c7 Sep 16 13:09:42.709841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.709753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmsv9" event={"ID":"90fdd31a-1830-4d96-8a94-f3083f6d9b85","Type":"ContainerStarted","Data":"ce67760d5505aca9452e93d88dd73e36fb38a5617ea9e0d75bd01b4d2f3e8f37"} Sep 16 13:09:42.712597 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.712567 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zs895" event={"ID":"906cc307-f99a-4be6-a72b-38024d094c15","Type":"ContainerStarted","Data":"df9daf9f926dd4d52125ef48ea9f09ec244e066149e5c73e8dbe61a855aa5b78"} Sep 16 13:09:42.712715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.712624 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zs895" event={"ID":"906cc307-f99a-4be6-a72b-38024d094c15","Type":"ContainerStarted","Data":"736509f4d90e2b1a8c3d240c5d2d872caee9cf7047f531251dd95600239cda71"} Sep 16 13:09:42.715655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.715631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzkfk" event={"ID":"1a8564d5-0e0d-4262-8b90-3af68cb8574e","Type":"ContainerStarted","Data":"894eb084d1f8db4c91716d2777adcdedc8df34db60d90cc3d9aca5ec934fafd6"} Sep 16 13:09:42.715769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.715689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzkfk" event={"ID":"1a8564d5-0e0d-4262-8b90-3af68cb8574e","Type":"ContainerStarted","Data":"9cdacd44674d78e4b1ac81aaa6b7531925e961ce1f6ba35aca9e79e6325780ac"} Sep 16 13:09:42.715769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.715722 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzkfk" event={"ID":"1a8564d5-0e0d-4262-8b90-3af68cb8574e","Type":"ContainerStarted","Data":"71f81917b17bc6bde7b4010ff36bc84198cd03ba227ce2599cf55e86af9653c7"} Sep 16 13:09:42.717988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.717966 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" event={"ID":"c9ce0f05-fb33-413b-be8d-11219809b2b9","Type":"ContainerStarted","Data":"4f3eec92e18dd02f4d0346b07b916a07006cda9a118e84cc6062c0fa0412d28f"} Sep 16 13:09:42.718084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.718008 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" event={"ID":"c9ce0f05-fb33-413b-be8d-11219809b2b9","Type":"ContainerStarted","Data":"8ba714b685e0e02ccd0844e2e972ca6119b12e9a9477ac75cd51c83b986e3f32"} Sep 16 13:09:42.718152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.718087 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:09:42.729931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.729879 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zs895" podStartSLOduration=66.532506147 podStartE2EDuration="1m7.729859572s" totalImagesPullingTime="1.197353425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:42.728799988 +0000 UTC m=+67.848526559" watchObservedRunningTime="2026-09-16 13:09:42.729859572 +0000 UTC m=+67.849586122" Sep 16 13:09:42.748915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:42.748860 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" podStartSLOduration=38.748843308 podStartE2EDuration="38.748843308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:42.747690485 +0000 UTC m=+67.867417058" watchObservedRunningTime="2026-09-16 13:09:42.748843308 +0000 UTC m=+67.868569859" Sep 16 13:09:43.149113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:43.149079 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:43.210798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:43.210765 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:45.606451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.606416 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8"] Sep 16 13:09:45.623416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.623332 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7"] Sep 16 13:09:45.623581 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.623547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.627413 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.627341 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 13:09:45.627590 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.627411 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 13:09:45.627739 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.627699 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ztjr6\"" Sep 16 13:09:45.639457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.639435 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-754f9b4486-mw6bn"] Sep 16 13:09:45.639566 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.639468 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw"] Sep 16 13:09:45.639629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.639573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.643614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.643595 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 13:09:45.644064 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.644046 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 13:09:45.644351 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.644332 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 13:09:45.645924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.645885 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 13:09:45.655606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.655580 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8"] Sep 16 13:09:45.655706 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.655656 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7"] Sep 16 13:09:45.655706 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.655678 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw"] Sep 16 13:09:45.655782 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.655756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.657957 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.657940 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4mfpp\"" Sep 16 13:09:45.658738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.658723 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 13:09:45.705804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.705327 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4"] Sep 16 13:09:45.731682 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.731651 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:45.734724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.734702 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zmxxp\"" Sep 16 13:09:45.735328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.735301 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 13:09:45.749259 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.749238 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4"] Sep 16 13:09:45.762505 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762482 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2sp\" (UniqueName: \"kubernetes.io/projected/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-kube-api-access-ql2sp\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.762629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-klusterlet-config\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.762629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8435540-7eea-40c4-a6ef-9395e7986931-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-h8ct8\" (UID: \"e8435540-7eea-40c4-a6ef-9395e7986931\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.762733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762637 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/364e1366-0816-4c29-a3e8-ac9c24fe6a37-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7875c44898-j79jw\" (UID: \"364e1366-0816-4c29-a3e8-ac9c24fe6a37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.762733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-tmp\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.762733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ks2l\" (UniqueName: \"kubernetes.io/projected/364e1366-0816-4c29-a3e8-ac9c24fe6a37-kube-api-access-6ks2l\") pod \"managed-serviceaccount-addon-agent-7875c44898-j79jw\" (UID: \"364e1366-0816-4c29-a3e8-ac9c24fe6a37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.762864 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.762787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e8435540-7eea-40c4-a6ef-9395e7986931-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-h8ct8\" (UID: \"e8435540-7eea-40c4-a6ef-9395e7986931\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.803954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.803927 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66f4746c99-qwlfr"] Sep 16 13:09:45.838260 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.838228 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66f4746c99-qwlfr"] Sep 16 13:09:45.838414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.838376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.863969 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.863905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-klusterlet-config\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.863969 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.863957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8435540-7eea-40c4-a6ef-9395e7986931-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-h8ct8\" (UID: \"e8435540-7eea-40c4-a6ef-9395e7986931\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.864196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.863994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e3d6c06b-3997-49de-baf6-6f7f1f8e8c85-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-gtrw4\" (UID: \"e3d6c06b-3997-49de-baf6-6f7f1f8e8c85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:45.864196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/364e1366-0816-4c29-a3e8-ac9c24fe6a37-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7875c44898-j79jw\" (UID: \"364e1366-0816-4c29-a3e8-ac9c24fe6a37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.864196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-tmp\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.864196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ks2l\" (UniqueName: \"kubernetes.io/projected/364e1366-0816-4c29-a3e8-ac9c24fe6a37-kube-api-access-6ks2l\") pod \"managed-serviceaccount-addon-agent-7875c44898-j79jw\" (UID: \"364e1366-0816-4c29-a3e8-ac9c24fe6a37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.864196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e8435540-7eea-40c4-a6ef-9395e7986931-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-h8ct8\" (UID: \"e8435540-7eea-40c4-a6ef-9395e7986931\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.864196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2sp\" (UniqueName: \"kubernetes.io/projected/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-kube-api-access-ql2sp\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.864843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864789 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-tmp\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.864936 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.864841 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8435540-7eea-40c4-a6ef-9395e7986931-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-h8ct8\" (UID: \"e8435540-7eea-40c4-a6ef-9395e7986931\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.867034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.867014 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e8435540-7eea-40c4-a6ef-9395e7986931-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-h8ct8\" (UID: \"e8435540-7eea-40c4-a6ef-9395e7986931\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.867206 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.867181 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-klusterlet-config\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.867319 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.867181 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/364e1366-0816-4c29-a3e8-ac9c24fe6a37-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7875c44898-j79jw\" (UID: \"364e1366-0816-4c29-a3e8-ac9c24fe6a37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.878902 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.878878 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2sp\" (UniqueName: \"kubernetes.io/projected/2a6f94d3-b3c7-4623-b7ee-8f323abfaa50-kube-api-access-ql2sp\") pod \"klusterlet-addon-workmgr-64d5bfbf55-2qfr7\" (UID: \"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.879063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.879043 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ks2l\" (UniqueName: \"kubernetes.io/projected/364e1366-0816-4c29-a3e8-ac9c24fe6a37-kube-api-access-6ks2l\") pod \"managed-serviceaccount-addon-agent-7875c44898-j79jw\" (UID: \"364e1366-0816-4c29-a3e8-ac9c24fe6a37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.935223 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.935190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" Sep 16 13:09:45.951402 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.951375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:45.965245 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-image-registry-private-configuration\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-trusted-ca\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-bound-sa-token\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnq2\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-kube-api-access-zgnq2\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-installation-pull-secrets\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e3d6c06b-3997-49de-baf6-6f7f1f8e8c85-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-gtrw4\" (UID: \"e3d6c06b-3997-49de-baf6-6f7f1f8e8c85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:45.965556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965471 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-ca-trust-extracted\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-registry-tls\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.965556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.965517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-registry-certificates\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:45.967731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.967711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e3d6c06b-3997-49de-baf6-6f7f1f8e8c85-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-gtrw4\" (UID: \"e3d6c06b-3997-49de-baf6-6f7f1f8e8c85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:45.972615 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.972594 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" Sep 16 13:09:45.974037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:45.974015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:46.041910 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.041875 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:46.066200 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-installation-pull-secrets\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066306 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-ca-trust-extracted\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066306 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-registry-tls\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066508 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-registry-certificates\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-image-registry-private-configuration\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066625 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-trusted-ca\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066625 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-bound-sa-token\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066723 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066616 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-ca-trust-extracted\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.066723 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.066643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnq2\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-kube-api-access-zgnq2\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.067349 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.067324 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-registry-certificates\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.067616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.067593 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-trusted-ca\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.068789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.068769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-installation-pull-secrets\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.068879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.068863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-image-registry-private-configuration\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.068940 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.068883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-registry-tls\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.070348 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.070318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:46.078321 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.078303 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnq2\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-kube-api-access-zgnq2\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.078728 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.078713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd7594e4-0006-4edd-8904-9ec96fa3ccaf-bound-sa-token\") pod \"image-registry-66f4746c99-qwlfr\" (UID: \"dd7594e4-0006-4edd-8904-9ec96fa3ccaf\") " pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.147716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.147693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.645908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.645849 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw"] Sep 16 13:09:46.649682 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:46.649653 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod364e1366_0816_4c29_a3e8_ac9c24fe6a37.slice/crio-bfc55357fe0eab52cfdbd2547210758ff675ba1c1273fd08e213a2cffd375da8 WatchSource:0}: Error finding container bfc55357fe0eab52cfdbd2547210758ff675ba1c1273fd08e213a2cffd375da8: Status 404 returned error can't find the container with id bfc55357fe0eab52cfdbd2547210758ff675ba1c1273fd08e213a2cffd375da8 Sep 16 13:09:46.665008 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.664939 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66f4746c99-qwlfr"] Sep 16 13:09:46.669415 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:46.669356 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7594e4_0006_4edd_8904_9ec96fa3ccaf.slice/crio-9d9d5fa3968d2129a79834937889aef592d7849a6c96e3d2e91af1a14ae5b8b3 WatchSource:0}: Error finding container 9d9d5fa3968d2129a79834937889aef592d7849a6c96e3d2e91af1a14ae5b8b3: Status 404 returned error can't find the container with id 9d9d5fa3968d2129a79834937889aef592d7849a6c96e3d2e91af1a14ae5b8b3 Sep 16 13:09:46.683106 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.683084 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8"] Sep 16 13:09:46.688443 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:46.688407 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8435540_7eea_40c4_a6ef_9395e7986931.slice/crio-bd856d0dfe9f75148b5eb4618300f2f9bd7fbb14201a8c4862541af2af72ce56 WatchSource:0}: Error finding container bd856d0dfe9f75148b5eb4618300f2f9bd7fbb14201a8c4862541af2af72ce56: Status 404 returned error can't find the container with id bd856d0dfe9f75148b5eb4618300f2f9bd7fbb14201a8c4862541af2af72ce56 Sep 16 13:09:46.694095 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.693511 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4"] Sep 16 13:09:46.699190 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.698663 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7"] Sep 16 13:09:46.706338 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:46.706310 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6f94d3_b3c7_4623_b7ee_8f323abfaa50.slice/crio-263ff6411dd97febc41cb567f0998ba141ca024ae8e03cbd4740ad9991ca0259 WatchSource:0}: Error finding container 263ff6411dd97febc41cb567f0998ba141ca024ae8e03cbd4740ad9991ca0259: Status 404 returned error can't find the container with id 263ff6411dd97febc41cb567f0998ba141ca024ae8e03cbd4740ad9991ca0259 Sep 16 13:09:46.733461 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.733431 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" event={"ID":"e8435540-7eea-40c4-a6ef-9395e7986931","Type":"ContainerStarted","Data":"bd856d0dfe9f75148b5eb4618300f2f9bd7fbb14201a8c4862541af2af72ce56"} Sep 16 13:09:46.734606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.734585 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" event={"ID":"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50","Type":"ContainerStarted","Data":"263ff6411dd97febc41cb567f0998ba141ca024ae8e03cbd4740ad9991ca0259"} Sep 16 13:09:46.735868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.735837 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" event={"ID":"e3d6c06b-3997-49de-baf6-6f7f1f8e8c85","Type":"ContainerStarted","Data":"f7e36666fbed56d60693191bdf6b141e2ed4544f2600a94eab064bb21f75c591"} Sep 16 13:09:46.737125 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.737104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" event={"ID":"364e1366-0816-4c29-a3e8-ac9c24fe6a37","Type":"ContainerStarted","Data":"bfc55357fe0eab52cfdbd2547210758ff675ba1c1273fd08e213a2cffd375da8"} Sep 16 13:09:46.738634 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.738611 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zvh5v" event={"ID":"b6dc3916-9abe-4021-a871-0deca7e008f1","Type":"ContainerStarted","Data":"06a511a07ea5975c0cbaa08ab2bcd694270507b03e3e42828775fb8591a4d2af"} Sep 16 13:09:46.741465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.741439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzkfk" event={"ID":"1a8564d5-0e0d-4262-8b90-3af68cb8574e","Type":"ContainerStarted","Data":"15e6f1c7354ee73e8ce99cd032b7245d53cffaa86186e54ba3749ba6d4f511c8"} Sep 16 13:09:46.743442 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.743405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" event={"ID":"dd7594e4-0006-4edd-8904-9ec96fa3ccaf","Type":"ContainerStarted","Data":"9d9d5fa3968d2129a79834937889aef592d7849a6c96e3d2e91af1a14ae5b8b3"} Sep 16 13:09:46.743547 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.743516 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:09:46.745665 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.745643 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmsv9" event={"ID":"90fdd31a-1830-4d96-8a94-f3083f6d9b85","Type":"ContainerStarted","Data":"0d666f4873bc910798c86ca3784b3889066a2c7512e5bff5ef80ba6f7289552d"} Sep 16 13:09:46.745747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.745689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmsv9" event={"ID":"90fdd31a-1830-4d96-8a94-f3083f6d9b85","Type":"ContainerStarted","Data":"38dc543ac2f4a77809847ce9efdb0988e6f741f099db8df590530be62b56b7ad"} Sep 16 13:09:46.745794 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.745778 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:46.767966 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.767892 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zvh5v" podStartSLOduration=65.98056911 podStartE2EDuration="1m11.767874329s" totalImagesPullingTime="5.787305219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:46.766952515 +0000 UTC m=+71.886679061" watchObservedRunningTime="2026-09-16 13:09:46.767874329 +0000 UTC m=+71.887600879" Sep 16 13:09:46.787814 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.787660 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" podStartSLOduration=1.787638995 podStartE2EDuration="1.787638995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:46.786434996 +0000 UTC m=+71.906161605" watchObservedRunningTime="2026-09-16 13:09:46.787638995 +0000 UTC m=+71.907365545" Sep 16 13:09:46.802788 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.802731 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lmsv9" podStartSLOduration=33.207177646 podStartE2EDuration="37.802711391s" totalImagesPullingTime="4.595533745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:46.802064711 +0000 UTC m=+71.921791278" watchObservedRunningTime="2026-09-16 13:09:46.802711391 +0000 UTC m=+71.922437942" Sep 16 13:09:46.821577 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:46.821518 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hzkfk" podStartSLOduration=33.581460742 podStartE2EDuration="37.821498209s" totalImagesPullingTime="4.240037467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:09:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:46.820647268 +0000 UTC m=+71.940373835" watchObservedRunningTime="2026-09-16 13:09:46.821498209 +0000 UTC m=+71.941224765" Sep 16 13:09:47.037593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:47.037555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:47.757379 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:47.756379 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" event={"ID":"dd7594e4-0006-4edd-8904-9ec96fa3ccaf","Type":"ContainerStarted","Data":"357e60debbfd39e0dd6d4c6213d6d8565d6ed5cc21a6da475ce877ee5f14b9ce"} Sep 16 13:09:49.408026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:49.407984 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:49.627467 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:49.627438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:52.776493 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.776443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" event={"ID":"e8435540-7eea-40c4-a6ef-9395e7986931","Type":"ContainerStarted","Data":"7333b4e408ddc36821b13e9161b584aa3374718bb664264f13fa14d4f6b64b3a"} Sep 16 13:09:52.777914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.777890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" event={"ID":"2a6f94d3-b3c7-4623-b7ee-8f323abfaa50","Type":"ContainerStarted","Data":"803436740f876790d97beb9cf55503425da98deafae133f308e248d0f0d52d29"} Sep 16 13:09:52.778038 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.778016 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:52.779515 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.779484 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" event={"ID":"e3d6c06b-3997-49de-baf6-6f7f1f8e8c85","Type":"ContainerStarted","Data":"e255b314e25e716548b2245fb157548cb77763397d9a6b45b77a95d387e16a13"} Sep 16 13:09:52.779628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.779548 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:52.779946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.779925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" Sep 16 13:09:52.781442 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.781423 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" event={"ID":"364e1366-0816-4c29-a3e8-ac9c24fe6a37","Type":"ContainerStarted","Data":"ae19b0655597492430c71e60735734a7dd165c2cffd4a384cb4cdf99b468a4e2"} Sep 16 13:09:52.784182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.784162 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" Sep 16 13:09:52.791855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.791817 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-h8ct8" podStartSLOduration=2.328442496 podStartE2EDuration="7.791805246s" totalImagesPullingTime="5.46336275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:52.790884385 +0000 UTC m=+77.910610943" watchObservedRunningTime="2026-09-16 13:09:52.791805246 +0000 UTC m=+77.911531832" Sep 16 13:09:52.806524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.806487 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7875c44898-j79jw" podStartSLOduration=2.304485422 podStartE2EDuration="7.806475133s" totalImagesPullingTime="5.501989711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:52.805462137 +0000 UTC m=+77.925188688" watchObservedRunningTime="2026-09-16 13:09:52.806475133 +0000 UTC m=+77.926201673" Sep 16 13:09:52.822652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.822612 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64d5bfbf55-2qfr7" podStartSLOduration=2.299391903 podStartE2EDuration="7.822599804s" totalImagesPullingTime="5.523207901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:52.821477882 +0000 UTC m=+77.941204447" watchObservedRunningTime="2026-09-16 13:09:52.822599804 +0000 UTC m=+77.942326353" Sep 16 13:09:52.837538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.837497 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-gtrw4" podStartSLOduration=2.390246843 podStartE2EDuration="7.837483796s" totalImagesPullingTime="5.447236953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:52.836604019 +0000 UTC m=+77.956330580" watchObservedRunningTime="2026-09-16 13:09:52.837483796 +0000 UTC m=+77.957210345" Sep 16 13:09:52.905785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:52.905751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:53.006350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:53.006308 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:53.295305 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:53.295259 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:53.372041 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:53.372006 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:53.513403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:53.513374 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:56.231512 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:56.231471 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:56.507918 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:56.507817 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:56.609099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:56.609049 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:56.709874 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:56.709845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:56.758408 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:56.758349 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lmsv9" Sep 16 13:09:58.450876 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.450754 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:58.664881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.664849 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:58.721844 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.721757 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dtm8g"] Sep 16 13:09:58.725274 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.725256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.728051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.728024 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 13:09:58.728416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.728392 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 13:09:58.728576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.728473 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 13:09:58.728641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.728582 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 13:09:58.729104 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.729087 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 13:09:58.729210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.729122 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 13:09:58.729210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.729122 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-djg9t\"" Sep 16 13:09:58.762752 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.762728 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:58.828785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.828761 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:58.866359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4591d7af-e2b3-4d00-b38d-2797cfc3a086-metrics-client-ca\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9flp\" (UniqueName: \"kubernetes.io/projected/4591d7af-e2b3-4d00-b38d-2797cfc3a086-kube-api-access-j9flp\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-tls\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-sys\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-wtmp\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-root\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.866683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.866679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-textfile\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.930012 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.929986 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:58.967801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.967770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.967930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.967807 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-wtmp\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.967930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.967843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-root\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.967930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.967884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.967930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.967921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-textfile\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.967960 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-wtmp\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968007 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-root\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4591d7af-e2b3-4d00-b38d-2797cfc3a086-metrics-client-ca\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968236 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9flp\" (UniqueName: \"kubernetes.io/projected/4591d7af-e2b3-4d00-b38d-2797cfc3a086-kube-api-access-j9flp\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968236 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968118 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-tls\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968236 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-sys\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968236 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968214 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4591d7af-e2b3-4d00-b38d-2797cfc3a086-sys\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968456 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:58.968309 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 13:09:58.968456 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:09:58.968370 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-tls podName:4591d7af-e2b3-4d00-b38d-2797cfc3a086 nodeName:}" failed. No retries permitted until 2026-09-16 13:09:59.468352934 +0000 UTC m=+84.588079464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-tls") pod "node-exporter-dtm8g" (UID: "4591d7af-e2b3-4d00-b38d-2797cfc3a086") : secret "node-exporter-tls" not found Sep 16 13:09:58.968567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968501 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-accelerators-collector-config\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968659 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4591d7af-e2b3-4d00-b38d-2797cfc3a086-metrics-client-ca\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.968775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.968720 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-textfile\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.970225 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.970203 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:58.976182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:58.976138 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9flp\" (UniqueName: \"kubernetes.io/projected/4591d7af-e2b3-4d00-b38d-2797cfc3a086-kube-api-access-j9flp\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:59.047902 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.047864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.112338 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.112304 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.242330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.242232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.411161 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.411116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.470960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.470933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-tls\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:59.473179 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.473157 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4591d7af-e2b3-4d00-b38d-2797cfc3a086-node-exporter-tls\") pod \"node-exporter-dtm8g\" (UID: \"4591d7af-e2b3-4d00-b38d-2797cfc3a086\") " pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:59.512961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.512898 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.609912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.609883 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.637270 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.637244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dtm8g" Sep 16 13:09:59.650848 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:09:59.650826 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4591d7af_e2b3_4d00_b38d_2797cfc3a086.slice/crio-f6931a8ffb2f375414a90835412b1383a4d64691e0701d711d82969c3cbb555e WatchSource:0}: Error finding container f6931a8ffb2f375414a90835412b1383a4d64691e0701d711d82969c3cbb555e: Status 404 returned error can't find the container with id f6931a8ffb2f375414a90835412b1383a4d64691e0701d711d82969c3cbb555e Sep 16 13:09:59.768063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.767989 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.806099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.806069 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtm8g" event={"ID":"4591d7af-e2b3-4d00-b38d-2797cfc3a086","Type":"ContainerStarted","Data":"f6931a8ffb2f375414a90835412b1383a4d64691e0701d711d82969c3cbb555e"} Sep 16 13:09:59.844206 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.844174 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:09:59.973510 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:09:59.973460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:00.074454 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:00.074367 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:00.811313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:00.811268 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtm8g" event={"ID":"4591d7af-e2b3-4d00-b38d-2797cfc3a086","Type":"ContainerStarted","Data":"5e3180ddff048fd4086a6844d968f63120f12245b0eda851cc8e2d6faa39fbda"} Sep 16 13:10:01.082708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.082624 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:01.303741 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.303705 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:01.398995 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.398911 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:01.619689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.619660 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:01.696082 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.695995 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:01.816446 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.816409 2581 generic.go:356] "Generic (PLEG): container finished" podID="4591d7af-e2b3-4d00-b38d-2797cfc3a086" containerID="5e3180ddff048fd4086a6844d968f63120f12245b0eda851cc8e2d6faa39fbda" exitCode=0 Sep 16 13:10:01.816816 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.816489 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtm8g" event={"ID":"4591d7af-e2b3-4d00-b38d-2797cfc3a086","Type":"ContainerDied","Data":"5e3180ddff048fd4086a6844d968f63120f12245b0eda851cc8e2d6faa39fbda"} Sep 16 13:10:01.831074 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:01.831045 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:02.081259 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:02.081225 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:02.264959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:02.264929 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:02.325019 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:02.324988 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:02.822053 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:02.822018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtm8g" event={"ID":"4591d7af-e2b3-4d00-b38d-2797cfc3a086","Type":"ContainerStarted","Data":"1c4f30fea7264f224080724847443ac9a6b7a7541b89671bb4d8bd4957a0e3bd"} Sep 16 13:10:02.822458 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:02.822115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dtm8g" event={"ID":"4591d7af-e2b3-4d00-b38d-2797cfc3a086","Type":"ContainerStarted","Data":"1961931037e010923f9d98f5119e67be0170ba8850dc81d6cc14bd0f21d1039a"} Sep 16 13:10:02.849195 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:02.849144 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dtm8g" podStartSLOduration=3.721183479 podStartE2EDuration="4.84912976s" totalImagesPullingTime="1.065775034s" totalInitContainerRuntime="62.171247ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:02.848545587 +0000 UTC m=+87.968272137" watchObservedRunningTime="2026-09-16 13:10:02.84912976 +0000 UTC m=+87.968856308" Sep 16 13:10:03.543458 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:03.543422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:03.703489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:03.703450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:03.842671 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:03.842583 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:05.034378 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:05.034342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:05.645028 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:05.644997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:10:06.091519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:06.091486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:06.215983 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:06.215935 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:07.731085 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:07.731049 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:07.851825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:07.851787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:08.234665 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:08.234628 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:08.388312 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:08.388267 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:08.764389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:08.764365 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66f4746c99-qwlfr" Sep 16 13:10:08.996397 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:08.996362 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:09.109017 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.108929 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:09.303417 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.303380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:09.590095 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.590063 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-869884d586-wqp2p"] Sep 16 13:10:09.594346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.594324 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.599296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.599259 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 13:10:09.599418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.599321 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 13:10:09.599418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.599372 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 13:10:09.599418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.599370 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 13:10:09.599928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.599910 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 13:10:09.600035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.600015 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 13:10:09.600401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.600388 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ldtfs\"" Sep 16 13:10:09.600461 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.600442 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 13:10:09.622645 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.622620 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869884d586-wqp2p"] Sep 16 13:10:09.749675 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.749641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-config\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.749837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.749693 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-oauth-serving-cert\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.749837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.749721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-serving-cert\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.749837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.749739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-oauth-config\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.749837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.749789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-service-ca\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.749837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.749822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhzx\" (UniqueName: \"kubernetes.io/projected/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-kube-api-access-snhzx\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.850556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.850471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhzx\" (UniqueName: \"kubernetes.io/projected/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-kube-api-access-snhzx\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.850556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.850525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-config\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851059 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.850580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-oauth-serving-cert\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851059 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.850613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-serving-cert\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851059 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.850664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-oauth-config\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851059 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.850722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-service-ca\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.851397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-config\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.851413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-oauth-serving-cert\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.851536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.851400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-service-ca\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.853061 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.853038 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-oauth-config\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.853186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.853167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-serving-cert\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.862702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.862679 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhzx\" (UniqueName: \"kubernetes.io/projected/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-kube-api-access-snhzx\") pod \"console-869884d586-wqp2p\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:09.903580 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:09.903552 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:10.179163 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.179089 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:10.192521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.192490 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869884d586-wqp2p"] Sep 16 13:10:10.199185 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:10:10.199166 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf028a13b_10e0_4a4f_8df8_23aa3cfacac6.slice/crio-6ad6c68a4aef49149c5f79ef2c8d753fe0974989385073547c75bfc68264ee01 WatchSource:0}: Error finding container 6ad6c68a4aef49149c5f79ef2c8d753fe0974989385073547c75bfc68264ee01: Status 404 returned error can't find the container with id 6ad6c68a4aef49149c5f79ef2c8d753fe0974989385073547c75bfc68264ee01 Sep 16 13:10:10.312572 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.312539 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:10.661711 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.661675 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" podUID="c9ce0f05-fb33-413b-be8d-11219809b2b9" containerName="registry" containerID="cri-o://4f3eec92e18dd02f4d0346b07b916a07006cda9a118e84cc6062c0fa0412d28f" gracePeriod=30 Sep 16 13:10:10.853783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.853740 2581 generic.go:356] "Generic (PLEG): container finished" podID="c9ce0f05-fb33-413b-be8d-11219809b2b9" containerID="4f3eec92e18dd02f4d0346b07b916a07006cda9a118e84cc6062c0fa0412d28f" exitCode=0 Sep 16 13:10:10.854218 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.853871 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" event={"ID":"c9ce0f05-fb33-413b-be8d-11219809b2b9","Type":"ContainerDied","Data":"4f3eec92e18dd02f4d0346b07b916a07006cda9a118e84cc6062c0fa0412d28f"} Sep 16 13:10:10.855677 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.855647 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869884d586-wqp2p" event={"ID":"f028a13b-10e0-4a4f-8df8-23aa3cfacac6","Type":"ContainerStarted","Data":"6ad6c68a4aef49149c5f79ef2c8d753fe0974989385073547c75bfc68264ee01"} Sep 16 13:10:10.912753 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:10.912686 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:10:11.059245 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059205 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-trusted-ca\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059303 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-certificates\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059331 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-installation-pull-secrets\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059363 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059390 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-image-registry-private-configuration\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059427 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-bound-sa-token\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059461 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9ce0f05-fb33-413b-be8d-11219809b2b9-ca-trust-extracted\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059498 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtlq4\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-kube-api-access-rtlq4\") pod \"c9ce0f05-fb33-413b-be8d-11219809b2b9\" (UID: \"c9ce0f05-fb33-413b-be8d-11219809b2b9\") " Sep 16 13:10:11.059697 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059668 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:10:11.059749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059714 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:10:11.059805 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.059751 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-trusted-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.062423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.062392 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:10:11.062533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.062421 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:10:11.062533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.062485 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:10:11.062604 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.062530 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-kube-api-access-rtlq4" (OuterVolumeSpecName: "kube-api-access-rtlq4") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "kube-api-access-rtlq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:10:11.063430 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.063404 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:10:11.071516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.071493 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ce0f05-fb33-413b-be8d-11219809b2b9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c9ce0f05-fb33-413b-be8d-11219809b2b9" (UID: "c9ce0f05-fb33-413b-be8d-11219809b2b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:10:11.161203 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161172 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlq4\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-kube-api-access-rtlq4\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.161203 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161198 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-certificates\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.161401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161211 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-installation-pull-secrets\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.161401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161226 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-registry-tls\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.161401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161243 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9ce0f05-fb33-413b-be8d-11219809b2b9-image-registry-private-configuration\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.161401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161259 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9ce0f05-fb33-413b-be8d-11219809b2b9-bound-sa-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.161401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.161272 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9ce0f05-fb33-413b-be8d-11219809b2b9-ca-trust-extracted\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:11.297056 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.297016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:11.399403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.399370 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:11.862907 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.862870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" event={"ID":"c9ce0f05-fb33-413b-be8d-11219809b2b9","Type":"ContainerDied","Data":"8ba714b685e0e02ccd0844e2e972ca6119b12e9a9477ac75cd51c83b986e3f32"} Sep 16 13:10:11.863382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.862947 2581 scope.go:117] "RemoveContainer" containerID="4f3eec92e18dd02f4d0346b07b916a07006cda9a118e84cc6062c0fa0412d28f" Sep 16 13:10:11.863382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.862881 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-754f9b4486-mw6bn" Sep 16 13:10:11.886112 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.886083 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-754f9b4486-mw6bn"] Sep 16 13:10:11.892005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:11.891983 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-754f9b4486-mw6bn"] Sep 16 13:10:12.721210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:12.721180 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f4nw4" Sep 16 13:10:13.385652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:13.385609 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ce0f05-fb33-413b-be8d-11219809b2b9" path="/var/lib/kubelet/pods/c9ce0f05-fb33-413b-be8d-11219809b2b9/volumes" Sep 16 13:10:13.872921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:13.872884 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869884d586-wqp2p" event={"ID":"f028a13b-10e0-4a4f-8df8-23aa3cfacac6","Type":"ContainerStarted","Data":"dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13"} Sep 16 13:10:13.893704 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:13.893656 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-869884d586-wqp2p" podStartSLOduration=2.20079403 podStartE2EDuration="4.893639467s" totalImagesPullingTime="2.692845437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:13.892051181 +0000 UTC m=+99.011777760" watchObservedRunningTime="2026-09-16 13:10:13.893639467 +0000 UTC m=+99.013366015" Sep 16 13:10:14.904574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:14.904545 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:15.004685 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:15.004655 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:15.563944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:15.563900 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:15.625905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:15.625862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:18.894465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.894434 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f96948-kmtg2"] Sep 16 13:10:18.894903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.894686 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ce0f05-fb33-413b-be8d-11219809b2b9" containerName="registry" Sep 16 13:10:18.894903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.894697 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ce0f05-fb33-413b-be8d-11219809b2b9" containerName="registry" Sep 16 13:10:18.894903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.894750 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ce0f05-fb33-413b-be8d-11219809b2b9" containerName="registry" Sep 16 13:10:18.929383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.929356 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f96948-kmtg2"] Sep 16 13:10:18.929564 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.929450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:18.937233 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:18.937205 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 13:10:19.020905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.020876 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-oauth-serving-cert\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.021003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.020908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-serving-cert\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.021003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.020928 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-service-ca\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.021003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.020958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-console-config\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.021003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.020978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-trusted-ca-bundle\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.021137 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.021003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjm9c\" (UniqueName: \"kubernetes.io/projected/703f541c-638e-4cf2-8af2-feb8251b580f-kube-api-access-gjm9c\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.021137 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.021022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-oauth-config\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.121865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121837 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-oauth-serving-cert\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.121963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-serving-cert\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.121963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-service-ca\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.121963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-console-config\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.121963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-trusted-ca-bundle\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.122135 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjm9c\" (UniqueName: \"kubernetes.io/projected/703f541c-638e-4cf2-8af2-feb8251b580f-kube-api-access-gjm9c\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.122135 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.121984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-oauth-config\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.122694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.122668 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-oauth-serving-cert\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.122806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.122707 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-console-config\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.122853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.122800 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-trusted-ca-bundle\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.122853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.122819 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-service-ca\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.124375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.124358 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-serving-cert\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.124519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.124500 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-oauth-config\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.130106 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.130086 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjm9c\" (UniqueName: \"kubernetes.io/projected/703f541c-638e-4cf2-8af2-feb8251b580f-kube-api-access-gjm9c\") pod \"console-7f96948-kmtg2\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.238310 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.238271 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:19.302806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.302778 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:19.403415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.403380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:19.525065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.525000 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f96948-kmtg2"] Sep 16 13:10:19.528616 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:10:19.528589 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703f541c_638e_4cf2_8af2_feb8251b580f.slice/crio-992fb68b8758acb5503246a43539ef7945624e37f31b7786f65d3f759c44b523 WatchSource:0}: Error finding container 992fb68b8758acb5503246a43539ef7945624e37f31b7786f65d3f759c44b523: Status 404 returned error can't find the container with id 992fb68b8758acb5503246a43539ef7945624e37f31b7786f65d3f759c44b523 Sep 16 13:10:19.899755 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.896094 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f96948-kmtg2" event={"ID":"703f541c-638e-4cf2-8af2-feb8251b580f","Type":"ContainerStarted","Data":"0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c"} Sep 16 13:10:19.899755 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.896190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f96948-kmtg2" event={"ID":"703f541c-638e-4cf2-8af2-feb8251b580f","Type":"ContainerStarted","Data":"992fb68b8758acb5503246a43539ef7945624e37f31b7786f65d3f759c44b523"} Sep 16 13:10:19.904404 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.904379 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:19.904536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.904429 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:19.908767 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.908747 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:19.916015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:19.915976 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f96948-kmtg2" podStartSLOduration=1.915963089 podStartE2EDuration="1.915963089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:19.914876804 +0000 UTC m=+105.034603390" watchObservedRunningTime="2026-09-16 13:10:19.915963089 +0000 UTC m=+105.035689638" Sep 16 13:10:20.903892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:20.903868 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:22.056089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:22.056056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:22.156386 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:22.156353 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:22.258073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:22.258035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:25.444231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:25.444194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:25.499210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:25.499179 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:26.903508 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:26.903474 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:27.003421 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:27.003379 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:27.711292 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:27.711261 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:27.813128 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:27.813098 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:27.926537 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:27.926506 2581 generic.go:356] "Generic (PLEG): container finished" podID="5072e389-8c92-4961-b1f8-8cdc1cc4fa33" containerID="b0a230ec0d2b704209fb077650947fd8f6538f135f9dfb5ac9c8564d8fe7547a" exitCode=0 Sep 16 13:10:27.926905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:27.926578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" event={"ID":"5072e389-8c92-4961-b1f8-8cdc1cc4fa33","Type":"ContainerDied","Data":"b0a230ec0d2b704209fb077650947fd8f6538f135f9dfb5ac9c8564d8fe7547a"} Sep 16 13:10:27.926964 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:27.926915 2581 scope.go:117] "RemoveContainer" containerID="b0a230ec0d2b704209fb077650947fd8f6538f135f9dfb5ac9c8564d8fe7547a" Sep 16 13:10:28.149348 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:28.149318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:28.250348 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:28.250252 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:28.932430 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:28.932397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jk6kn" event={"ID":"5072e389-8c92-4961-b1f8-8cdc1cc4fa33","Type":"ContainerStarted","Data":"9b94e632ef5583a43ea67688e30a57e2ea032f6f7cad9668c6cc1f42710418a5"} Sep 16 13:10:29.238763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:29.238729 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:29.238953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:29.238796 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:29.243987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:29.243959 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:29.939630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:29.939600 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:10:29.982186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:29.982159 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869884d586-wqp2p"] Sep 16 13:10:30.085037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:30.085000 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:30.144585 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:30.144532 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:30.309773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:30.309742 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:43.154502 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:43.154460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:43.208857 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:43.208823 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:48.098198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:48.098159 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:48.198142 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:48.198107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:10:55.001633 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.001566 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-869884d586-wqp2p" podUID="f028a13b-10e0-4a4f-8df8-23aa3cfacac6" containerName="console" containerID="cri-o://dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13" gracePeriod=15 Sep 16 13:10:55.236338 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.236313 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:55.387504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387442 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-oauth-config\") pod \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " Sep 16 13:10:55.387504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387472 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-serving-cert\") pod \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " Sep 16 13:10:55.387504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387491 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-oauth-serving-cert\") pod \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " Sep 16 13:10:55.387762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387535 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-service-ca\") pod \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " Sep 16 13:10:55.387762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387558 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-config\") pod \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " Sep 16 13:10:55.387762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387575 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snhzx\" (UniqueName: \"kubernetes.io/projected/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-kube-api-access-snhzx\") pod \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\" (UID: \"f028a13b-10e0-4a4f-8df8-23aa3cfacac6\") " Sep 16 13:10:55.387980 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387953 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-service-ca" (OuterVolumeSpecName: "service-ca") pod "f028a13b-10e0-4a4f-8df8-23aa3cfacac6" (UID: "f028a13b-10e0-4a4f-8df8-23aa3cfacac6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:10:55.387980 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387967 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f028a13b-10e0-4a4f-8df8-23aa3cfacac6" (UID: "f028a13b-10e0-4a4f-8df8-23aa3cfacac6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:10:55.388092 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.387957 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-config" (OuterVolumeSpecName: "console-config") pod "f028a13b-10e0-4a4f-8df8-23aa3cfacac6" (UID: "f028a13b-10e0-4a4f-8df8-23aa3cfacac6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:10:55.389780 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.389756 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f028a13b-10e0-4a4f-8df8-23aa3cfacac6" (UID: "f028a13b-10e0-4a4f-8df8-23aa3cfacac6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:10:55.391323 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.389810 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-kube-api-access-snhzx" (OuterVolumeSpecName: "kube-api-access-snhzx") pod "f028a13b-10e0-4a4f-8df8-23aa3cfacac6" (UID: "f028a13b-10e0-4a4f-8df8-23aa3cfacac6"). InnerVolumeSpecName "kube-api-access-snhzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:10:55.391323 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.390041 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f028a13b-10e0-4a4f-8df8-23aa3cfacac6" (UID: "f028a13b-10e0-4a4f-8df8-23aa3cfacac6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:10:55.488735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.488690 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-oauth-config\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:55.488735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.488730 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-serving-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:55.488735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.488741 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-oauth-serving-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:55.488735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.488751 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:55.488988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.488760 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-console-config\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:55.488988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:55.488768 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhzx\" (UniqueName: \"kubernetes.io/projected/f028a13b-10e0-4a4f-8df8-23aa3cfacac6-kube-api-access-snhzx\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:10:56.027922 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.027883 2581 generic.go:356] "Generic (PLEG): container finished" podID="f028a13b-10e0-4a4f-8df8-23aa3cfacac6" containerID="dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13" exitCode=0 Sep 16 13:10:56.028373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.027938 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869884d586-wqp2p" event={"ID":"f028a13b-10e0-4a4f-8df8-23aa3cfacac6","Type":"ContainerDied","Data":"dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13"} Sep 16 13:10:56.028373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.027950 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869884d586-wqp2p" Sep 16 13:10:56.028373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.028023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869884d586-wqp2p" event={"ID":"f028a13b-10e0-4a4f-8df8-23aa3cfacac6","Type":"ContainerDied","Data":"6ad6c68a4aef49149c5f79ef2c8d753fe0974989385073547c75bfc68264ee01"} Sep 16 13:10:56.028373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.028051 2581 scope.go:117] "RemoveContainer" containerID="dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13" Sep 16 13:10:56.036767 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.036742 2581 scope.go:117] "RemoveContainer" containerID="dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13" Sep 16 13:10:56.037037 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:10:56.037015 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13\": container with ID starting with dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13 not found: ID does not exist" containerID="dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13" Sep 16 13:10:56.037101 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.037049 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13"} err="failed to get container status \"dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13\": rpc error: code = NotFound desc = could not find container \"dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13\": container with ID starting with dc96cbdf6edf4d366db3b3a0a7579595f7e455b3ff54230cd138233023ca5e13 not found: ID does not exist" Sep 16 13:10:56.047703 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.047681 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869884d586-wqp2p"] Sep 16 13:10:56.053194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:56.053176 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-869884d586-wqp2p"] Sep 16 13:10:57.387747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:10:57.387716 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f028a13b-10e0-4a4f-8df8-23aa3cfacac6" path="/var/lib/kubelet/pods/f028a13b-10e0-4a4f-8df8-23aa3cfacac6/volumes" Sep 16 13:11:14.798843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:14.798810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:14.861572 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:14.861537 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:26.720827 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:26.720791 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:26.768428 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:26.768396 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:43.162437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:43.162404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:43.262761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:43.262728 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:53.110411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.110380 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf5d44cb9-tphb5"] Sep 16 13:11:53.110794 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.110664 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f028a13b-10e0-4a4f-8df8-23aa3cfacac6" containerName="console" Sep 16 13:11:53.110794 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.110675 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f028a13b-10e0-4a4f-8df8-23aa3cfacac6" containerName="console" Sep 16 13:11:53.110794 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.110722 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f028a13b-10e0-4a4f-8df8-23aa3cfacac6" containerName="console" Sep 16 13:11:53.114666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.114646 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.124396 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.124378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf5d44cb9-tphb5"] Sep 16 13:11:53.166016 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.165984 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:53.208814 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.208779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-config\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.208814 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.208813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-service-ca\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.209000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.208846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-serving-cert\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.209000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.208916 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrwj\" (UniqueName: \"kubernetes.io/projected/0cb5e70c-bee6-4e63-8d74-718b33263fa4-kube-api-access-fxrwj\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.209000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.208948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-oauth-config\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.209000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.208977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-trusted-ca-bundle\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.209125 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.209003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-oauth-serving-cert\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.220225 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.220200 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:53.308838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.308812 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:53.309782 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309760 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-serving-cert\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.309882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrwj\" (UniqueName: \"kubernetes.io/projected/0cb5e70c-bee6-4e63-8d74-718b33263fa4-kube-api-access-fxrwj\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.309882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-oauth-config\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.309882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-trusted-ca-bundle\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.309882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-oauth-serving-cert\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.310076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-config\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.310076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.309913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-service-ca\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.310731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.310706 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-oauth-serving-cert\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.310842 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.310787 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-service-ca\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.310907 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.310865 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-config\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.310907 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.310873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-trusted-ca-bundle\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.312305 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.312263 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-serving-cert\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.312400 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.312384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-oauth-config\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.319314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.319277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrwj\" (UniqueName: \"kubernetes.io/projected/0cb5e70c-bee6-4e63-8d74-718b33263fa4-kube-api-access-fxrwj\") pod \"console-7bf5d44cb9-tphb5\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.411352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.411266 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:53.423628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.423612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:11:53.693792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.693750 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf5d44cb9-tphb5"] Sep 16 13:11:53.696049 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:11:53.696018 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb5e70c_bee6_4e63_8d74_718b33263fa4.slice/crio-0db7220f11e941310910def4722514d7cf3394162067449001dff1baf810914f WatchSource:0}: Error finding container 0db7220f11e941310910def4722514d7cf3394162067449001dff1baf810914f: Status 404 returned error can't find the container with id 0db7220f11e941310910def4722514d7cf3394162067449001dff1baf810914f Sep 16 13:11:53.918794 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.918751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:53.974134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:53.974105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:54.233512 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:54.233433 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf5d44cb9-tphb5" event={"ID":"0cb5e70c-bee6-4e63-8d74-718b33263fa4","Type":"ContainerStarted","Data":"49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47"} Sep 16 13:11:54.233512 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:54.233485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf5d44cb9-tphb5" event={"ID":"0cb5e70c-bee6-4e63-8d74-718b33263fa4","Type":"ContainerStarted","Data":"0db7220f11e941310910def4722514d7cf3394162067449001dff1baf810914f"} Sep 16 13:11:54.251875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:54.251830 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf5d44cb9-tphb5" podStartSLOduration=1.251814664 podStartE2EDuration="1.251814664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:11:54.251005757 +0000 UTC m=+199.370732305" watchObservedRunningTime="2026-09-16 13:11:54.251814664 +0000 UTC m=+199.371541212" Sep 16 13:11:54.518682 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:54.518598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:54.619833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:54.619795 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:56.787347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:56.787305 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:11:56.886250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:11:56.886212 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:03.424258 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:03.424227 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:12:03.424707 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:03.424275 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:12:03.428627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:03.428606 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:12:04.272320 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:04.272276 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:12:04.308680 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:04.308646 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f96948-kmtg2"] Sep 16 13:12:04.408687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:04.408643 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:04.517646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:04.517583 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:04.669359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:04.669264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:23.844496 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:23.844460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:29.327673 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.327627 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f96948-kmtg2" podUID="703f541c-638e-4cf2-8af2-feb8251b580f" containerName="console" containerID="cri-o://0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c" gracePeriod=15 Sep 16 13:12:29.562990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.562966 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:12:29.676666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676589 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-console-config\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.676666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676627 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-service-ca\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.676881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676667 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjm9c\" (UniqueName: \"kubernetes.io/projected/703f541c-638e-4cf2-8af2-feb8251b580f-kube-api-access-gjm9c\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.676881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676699 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-oauth-serving-cert\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.676881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676714 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-trusted-ca-bundle\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.676881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676765 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-serving-cert\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.676881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.676789 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-oauth-config\") pod \"703f541c-638e-4cf2-8af2-feb8251b580f\" (UID: \"703f541c-638e-4cf2-8af2-feb8251b580f\") " Sep 16 13:12:29.677147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.677119 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-service-ca" (OuterVolumeSpecName: "service-ca") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:12:29.677216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.677129 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-console-config" (OuterVolumeSpecName: "console-config") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:12:29.677216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.677172 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:12:29.677216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.677193 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:12:29.678955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.678922 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703f541c-638e-4cf2-8af2-feb8251b580f-kube-api-access-gjm9c" (OuterVolumeSpecName: "kube-api-access-gjm9c") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "kube-api-access-gjm9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:12:29.679047 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.678956 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:12:29.679047 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.678981 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "703f541c-638e-4cf2-8af2-feb8251b580f" (UID: "703f541c-638e-4cf2-8af2-feb8251b580f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:12:29.777627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777603 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-serving-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:29.777627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777623 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/703f541c-638e-4cf2-8af2-feb8251b580f-console-oauth-config\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:29.777770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777632 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-console-config\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:29.777770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777641 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:29.777770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777649 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjm9c\" (UniqueName: \"kubernetes.io/projected/703f541c-638e-4cf2-8af2-feb8251b580f-kube-api-access-gjm9c\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:29.777770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777658 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-oauth-serving-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:29.777770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:29.777666 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/703f541c-638e-4cf2-8af2-feb8251b580f-trusted-ca-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:12:30.364692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.364654 2581 generic.go:356] "Generic (PLEG): container finished" podID="703f541c-638e-4cf2-8af2-feb8251b580f" containerID="0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c" exitCode=0 Sep 16 13:12:30.365070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.364701 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f96948-kmtg2" Sep 16 13:12:30.365070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.364741 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f96948-kmtg2" event={"ID":"703f541c-638e-4cf2-8af2-feb8251b580f","Type":"ContainerDied","Data":"0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c"} Sep 16 13:12:30.365070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.364793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f96948-kmtg2" event={"ID":"703f541c-638e-4cf2-8af2-feb8251b580f","Type":"ContainerDied","Data":"992fb68b8758acb5503246a43539ef7945624e37f31b7786f65d3f759c44b523"} Sep 16 13:12:30.365070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.364815 2581 scope.go:117] "RemoveContainer" containerID="0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c" Sep 16 13:12:30.372843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.372818 2581 scope.go:117] "RemoveContainer" containerID="0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c" Sep 16 13:12:30.373114 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:12:30.373091 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c\": container with ID starting with 0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c not found: ID does not exist" containerID="0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c" Sep 16 13:12:30.373197 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.373122 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c"} err="failed to get container status \"0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c\": rpc error: code = NotFound desc = could not find container \"0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c\": container with ID starting with 0dc28a4a40c26feea5c27cc6105e19cc65e7f47f202c811e58fc53c91921197c not found: ID does not exist" Sep 16 13:12:30.384334 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.384310 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f96948-kmtg2"] Sep 16 13:12:30.389185 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:30.389166 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f96948-kmtg2"] Sep 16 13:12:31.385155 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:31.385112 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="703f541c-638e-4cf2-8af2-feb8251b580f" path="/var/lib/kubelet/pods/703f541c-638e-4cf2-8af2-feb8251b580f/volumes" Sep 16 13:12:44.682376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:44.682343 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:44.779702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:44.779654 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:12:45.493754 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.493719 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc"] Sep 16 13:12:45.493968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.493957 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703f541c-638e-4cf2-8af2-feb8251b580f" containerName="console" Sep 16 13:12:45.494012 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.493971 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="703f541c-638e-4cf2-8af2-feb8251b580f" containerName="console" Sep 16 13:12:45.494053 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.494025 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="703f541c-638e-4cf2-8af2-feb8251b580f" containerName="console" Sep 16 13:12:45.498558 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.498543 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.500949 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.500927 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:12:45.501626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.501612 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:12:45.501701 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.501611 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bjxjv\"" Sep 16 13:12:45.510929 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.510904 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc"] Sep 16 13:12:45.579916 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.579889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.580035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.579938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5tc\" (UniqueName: \"kubernetes.io/projected/f29246a7-ab5d-4853-96d5-38f62ec845e8-kube-api-access-7l5tc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.580035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.579966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.680576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.680552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.680732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.680592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5tc\" (UniqueName: \"kubernetes.io/projected/f29246a7-ab5d-4853-96d5-38f62ec845e8-kube-api-access-7l5tc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.680732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.680613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.680987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.680967 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.681047 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.680997 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.689641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.689620 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5tc\" (UniqueName: \"kubernetes.io/projected/f29246a7-ab5d-4853-96d5-38f62ec845e8-kube-api-access-7l5tc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:45.807519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:45.807466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:12:46.077132 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:46.077053 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc"] Sep 16 13:12:46.079899 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:12:46.079872 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29246a7_ab5d_4853_96d5_38f62ec845e8.slice/crio-0af189ca86870820f76d90597fe59a3ef044bf60e2f4e63bc9c969ed961ad307 WatchSource:0}: Error finding container 0af189ca86870820f76d90597fe59a3ef044bf60e2f4e63bc9c969ed961ad307: Status 404 returned error can't find the container with id 0af189ca86870820f76d90597fe59a3ef044bf60e2f4e63bc9c969ed961ad307 Sep 16 13:12:46.419773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:46.419693 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" event={"ID":"f29246a7-ab5d-4853-96d5-38f62ec845e8","Type":"ContainerStarted","Data":"0af189ca86870820f76d90597fe59a3ef044bf60e2f4e63bc9c969ed961ad307"} Sep 16 13:12:53.448156 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:53.448121 2581 generic.go:356] "Generic (PLEG): container finished" podID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerID="2aafcc7494e9c572cdeeecfda53fab7e7129a05cf71e9670b5ff62d680154ce4" exitCode=0 Sep 16 13:12:53.448554 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:53.448205 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" event={"ID":"f29246a7-ab5d-4853-96d5-38f62ec845e8","Type":"ContainerDied","Data":"2aafcc7494e9c572cdeeecfda53fab7e7129a05cf71e9670b5ff62d680154ce4"} Sep 16 13:12:56.461553 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:56.461516 2581 generic.go:356] "Generic (PLEG): container finished" podID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerID="e19c5cd81e5d65787d7a7ca9acdf7598fa993e1501729b5f9ea4b7e72c456f43" exitCode=0 Sep 16 13:12:56.461928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:12:56.461613 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" event={"ID":"f29246a7-ab5d-4853-96d5-38f62ec845e8","Type":"ContainerDied","Data":"e19c5cd81e5d65787d7a7ca9acdf7598fa993e1501729b5f9ea4b7e72c456f43"} Sep 16 13:13:00.478065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:00.477968 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:03.490545 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:03.490510 2581 generic.go:356] "Generic (PLEG): container finished" podID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerID="6961f88cad19bf20acb04e4787b7f9311a574d3cf7c7ceaadf3c77f117a48f73" exitCode=0 Sep 16 13:13:03.490853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:03.490580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" event={"ID":"f29246a7-ab5d-4853-96d5-38f62ec845e8","Type":"ContainerDied","Data":"6961f88cad19bf20acb04e4787b7f9311a574d3cf7c7ceaadf3c77f117a48f73"} Sep 16 13:13:04.613026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.613001 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:13:04.640232 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.640202 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-bundle\") pod \"f29246a7-ab5d-4853-96d5-38f62ec845e8\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " Sep 16 13:13:04.640232 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.640236 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l5tc\" (UniqueName: \"kubernetes.io/projected/f29246a7-ab5d-4853-96d5-38f62ec845e8-kube-api-access-7l5tc\") pod \"f29246a7-ab5d-4853-96d5-38f62ec845e8\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " Sep 16 13:13:04.640438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.640272 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-util\") pod \"f29246a7-ab5d-4853-96d5-38f62ec845e8\" (UID: \"f29246a7-ab5d-4853-96d5-38f62ec845e8\") " Sep 16 13:13:04.640906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.640881 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-bundle" (OuterVolumeSpecName: "bundle") pod "f29246a7-ab5d-4853-96d5-38f62ec845e8" (UID: "f29246a7-ab5d-4853-96d5-38f62ec845e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:04.642372 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.642353 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29246a7-ab5d-4853-96d5-38f62ec845e8-kube-api-access-7l5tc" (OuterVolumeSpecName: "kube-api-access-7l5tc") pod "f29246a7-ab5d-4853-96d5-38f62ec845e8" (UID: "f29246a7-ab5d-4853-96d5-38f62ec845e8"). InnerVolumeSpecName "kube-api-access-7l5tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:13:04.644244 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.644223 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-util" (OuterVolumeSpecName: "util") pod "f29246a7-ab5d-4853-96d5-38f62ec845e8" (UID: "f29246a7-ab5d-4853-96d5-38f62ec845e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:04.741052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.741005 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:04.741052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.741045 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f29246a7-ab5d-4853-96d5-38f62ec845e8-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:04.741052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:04.741054 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5tc\" (UniqueName: \"kubernetes.io/projected/f29246a7-ab5d-4853-96d5-38f62ec845e8-kube-api-access-7l5tc\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:05.499592 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:05.499562 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" Sep 16 13:13:05.499592 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:05.499573 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pqzvc" event={"ID":"f29246a7-ab5d-4853-96d5-38f62ec845e8","Type":"ContainerDied","Data":"0af189ca86870820f76d90597fe59a3ef044bf60e2f4e63bc9c969ed961ad307"} Sep 16 13:13:05.499799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:05.499614 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af189ca86870820f76d90597fe59a3ef044bf60e2f4e63bc9c969ed961ad307" Sep 16 13:13:06.561014 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:06.560980 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:08.340110 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:08.340073 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:10.175380 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:10.175342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:10.275542 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:10.275507 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:12.853460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:12.853422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:12.962246 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:12.962207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:13.522762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.522730 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:13.681252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.681213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:13.892330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892243 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq"] Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892518 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="util" Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892529 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="util" Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892543 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="pull" Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892549 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="pull" Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892559 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="extract" Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892564 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="extract" Sep 16 13:13:13.892669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.892610 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29246a7-ab5d-4853-96d5-38f62ec845e8" containerName="extract" Sep 16 13:13:13.895795 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.895778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.897035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.897014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.897147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.897056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrmk\" (UniqueName: \"kubernetes.io/projected/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-kube-api-access-5mrmk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.897147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.897095 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.897958 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.897944 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:13:13.897999 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.897944 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bjxjv\"" Sep 16 13:13:13.907562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.907546 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:13:13.916088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.916066 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq"] Sep 16 13:13:13.997619 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.997579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrmk\" (UniqueName: \"kubernetes.io/projected/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-kube-api-access-5mrmk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.997792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.997627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.997792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.997652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.997979 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.997964 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:13.998069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:13.998052 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:14.025724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:14.025693 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrmk\" (UniqueName: \"kubernetes.io/projected/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-kube-api-access-5mrmk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:14.204311 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:14.204265 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:14.480698 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:14.480622 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq"] Sep 16 13:13:14.483614 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:13:14.483586 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb07acd4_5de3_4e91_bf44_d5062ec7c7c8.slice/crio-def7f55aa7a6d0713c6006263accd459e652bb0ce37dd87e9cb65ea0aaa322a4 WatchSource:0}: Error finding container def7f55aa7a6d0713c6006263accd459e652bb0ce37dd87e9cb65ea0aaa322a4: Status 404 returned error can't find the container with id def7f55aa7a6d0713c6006263accd459e652bb0ce37dd87e9cb65ea0aaa322a4 Sep 16 13:13:14.531159 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:14.531128 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" event={"ID":"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8","Type":"ContainerStarted","Data":"def7f55aa7a6d0713c6006263accd459e652bb0ce37dd87e9cb65ea0aaa322a4"} Sep 16 13:13:15.260633 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:15.260603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:15.429504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:15.429409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:15.536452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:15.536419 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerID="cd7277ea2854a26f539a3e7b18e50f0b6f053062d496a763d1345c8701836bb1" exitCode=0 Sep 16 13:13:15.536589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:15.536508 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" event={"ID":"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8","Type":"ContainerDied","Data":"cd7277ea2854a26f539a3e7b18e50f0b6f053062d496a763d1345c8701836bb1"} Sep 16 13:13:16.973736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:16.973693 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:17.045389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:17.045351 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:18.549883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:18.549797 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerID="fc1ddc831caacaf8bee492ccc9939554bc6f98610aa1b70acf5ccc685ab3e93a" exitCode=0 Sep 16 13:13:18.549883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:18.549867 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" event={"ID":"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8","Type":"ContainerDied","Data":"fc1ddc831caacaf8bee492ccc9939554bc6f98610aa1b70acf5ccc685ab3e93a"} Sep 16 13:13:19.556063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:19.556027 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerID="4cda3de7cf492d3018d1257fa1d2555a833bc39fa2e909e7f5227698387df77e" exitCode=0 Sep 16 13:13:19.556455 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:19.556064 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" event={"ID":"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8","Type":"ContainerDied","Data":"4cda3de7cf492d3018d1257fa1d2555a833bc39fa2e909e7f5227698387df77e"} Sep 16 13:13:20.683884 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.683862 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:20.750728 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.750700 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-bundle\") pod \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " Sep 16 13:13:20.750843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.750777 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mrmk\" (UniqueName: \"kubernetes.io/projected/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-kube-api-access-5mrmk\") pod \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " Sep 16 13:13:20.750843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.750810 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-util\") pod \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\" (UID: \"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8\") " Sep 16 13:13:20.751119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.751097 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-bundle" (OuterVolumeSpecName: "bundle") pod "bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" (UID: "bb07acd4-5de3-4e91-bf44-d5062ec7c7c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:20.752872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.752852 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-kube-api-access-5mrmk" (OuterVolumeSpecName: "kube-api-access-5mrmk") pod "bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" (UID: "bb07acd4-5de3-4e91-bf44-d5062ec7c7c8"). InnerVolumeSpecName "kube-api-access-5mrmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:13:20.755363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.755341 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-util" (OuterVolumeSpecName: "util") pod "bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" (UID: "bb07acd4-5de3-4e91-bf44-d5062ec7c7c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:20.851623 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.851567 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrmk\" (UniqueName: \"kubernetes.io/projected/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-kube-api-access-5mrmk\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:20.851623 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.851591 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:20.851623 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:20.851601 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb07acd4-5de3-4e91-bf44-d5062ec7c7c8-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:21.564935 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:21.564897 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" event={"ID":"bb07acd4-5de3-4e91-bf44-d5062ec7c7c8","Type":"ContainerDied","Data":"def7f55aa7a6d0713c6006263accd459e652bb0ce37dd87e9cb65ea0aaa322a4"} Sep 16 13:13:21.564935 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:21.564932 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs76gq" Sep 16 13:13:21.564935 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:21.564945 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="def7f55aa7a6d0713c6006263accd459e652bb0ce37dd87e9cb65ea0aaa322a4" Sep 16 13:13:23.031015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:23.030978 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:24.415885 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.415851 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sdtxn"] Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416118 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="pull" Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416132 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="pull" Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416155 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="util" Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416160 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="util" Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416167 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="extract" Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416172 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="extract" Sep 16 13:13:24.416238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.416219 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb07acd4-5de3-4e91-bf44-d5062ec7c7c8" containerName="extract" Sep 16 13:13:24.420051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.420035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.422780 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.422693 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 13:13:24.423378 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.423356 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 13:13:24.423869 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.423845 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pg8hp\"" Sep 16 13:13:24.431010 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.430984 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sdtxn"] Sep 16 13:13:24.476666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.476635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjw7\" (UniqueName: \"kubernetes.io/projected/fd933e84-75bc-422e-8954-262788a849f7-kube-api-access-krjw7\") pod \"cert-manager-55b4d954d5-sdtxn\" (UID: \"fd933e84-75bc-422e-8954-262788a849f7\") " pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.476981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.476943 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd933e84-75bc-422e-8954-262788a849f7-bound-sa-token\") pod \"cert-manager-55b4d954d5-sdtxn\" (UID: \"fd933e84-75bc-422e-8954-262788a849f7\") " pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.577990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.577964 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd933e84-75bc-422e-8954-262788a849f7-bound-sa-token\") pod \"cert-manager-55b4d954d5-sdtxn\" (UID: \"fd933e84-75bc-422e-8954-262788a849f7\") " pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.578111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.578016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjw7\" (UniqueName: \"kubernetes.io/projected/fd933e84-75bc-422e-8954-262788a849f7-kube-api-access-krjw7\") pod \"cert-manager-55b4d954d5-sdtxn\" (UID: \"fd933e84-75bc-422e-8954-262788a849f7\") " pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.586488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.586464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjw7\" (UniqueName: \"kubernetes.io/projected/fd933e84-75bc-422e-8954-262788a849f7-kube-api-access-krjw7\") pod \"cert-manager-55b4d954d5-sdtxn\" (UID: \"fd933e84-75bc-422e-8954-262788a849f7\") " pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.586584 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.586465 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd933e84-75bc-422e-8954-262788a849f7-bound-sa-token\") pod \"cert-manager-55b4d954d5-sdtxn\" (UID: \"fd933e84-75bc-422e-8954-262788a849f7\") " pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.730858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.730833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sdtxn" Sep 16 13:13:24.998028 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:24.997948 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sdtxn"] Sep 16 13:13:25.000934 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:13:25.000909 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd933e84_75bc_422e_8954_262788a849f7.slice/crio-1c9ca8adbd9546dc17de99860ee7242ba5386cfeade173a221b8b3c00c32d864 WatchSource:0}: Error finding container 1c9ca8adbd9546dc17de99860ee7242ba5386cfeade173a221b8b3c00c32d864: Status 404 returned error can't find the container with id 1c9ca8adbd9546dc17de99860ee7242ba5386cfeade173a221b8b3c00c32d864 Sep 16 13:13:25.579719 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:25.579681 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sdtxn" event={"ID":"fd933e84-75bc-422e-8954-262788a849f7","Type":"ContainerStarted","Data":"1c9ca8adbd9546dc17de99860ee7242ba5386cfeade173a221b8b3c00c32d864"} Sep 16 13:13:26.904427 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:26.904378 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:27.006778 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:27.006742 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:28.250250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.250216 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b"] Sep 16 13:13:28.253115 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.253099 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.255563 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.255538 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:13:28.255666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.255644 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-x4xkb\"" Sep 16 13:13:28.256474 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.256456 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 13:13:28.262548 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.262529 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b"] Sep 16 13:13:28.305553 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.305529 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p677x\" (UniqueName: \"kubernetes.io/projected/c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1-kube-api-access-p677x\") pod \"openshift-lws-operator-fd8ccff4c-bfz7b\" (UID: \"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.305649 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.305561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bfz7b\" (UID: \"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.406547 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.406519 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bfz7b\" (UID: \"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.406656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.406603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p677x\" (UniqueName: \"kubernetes.io/projected/c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1-kube-api-access-p677x\") pod \"openshift-lws-operator-fd8ccff4c-bfz7b\" (UID: \"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.406865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.406846 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bfz7b\" (UID: \"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.418532 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.418506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p677x\" (UniqueName: \"kubernetes.io/projected/c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1-kube-api-access-p677x\") pod \"openshift-lws-operator-fd8ccff4c-bfz7b\" (UID: \"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.562596 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.562505 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" Sep 16 13:13:28.592596 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.592561 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sdtxn" event={"ID":"fd933e84-75bc-422e-8954-262788a849f7","Type":"ContainerStarted","Data":"071b721850bb5a9aa98700a6ad51d170f8fbc51971863b390954d919f099c780"} Sep 16 13:13:28.610224 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.610173 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-sdtxn" podStartSLOduration=1.983333922 podStartE2EDuration="4.610153223s" totalImagesPullingTime="2.626819301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:28.609768326 +0000 UTC m=+293.729494876" watchObservedRunningTime="2026-09-16 13:13:28.610153223 +0000 UTC m=+293.729879775" Sep 16 13:13:28.833475 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:28.833106 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b"] Sep 16 13:13:28.835662 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:13:28.835634 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9afd9e0_93e5_42f7_9a4e_5558e89fc6c1.slice/crio-8aee71c81cea5a6d1621249d46ef5b2cef915d8723cb9bfcce5d379d381e2219 WatchSource:0}: Error finding container 8aee71c81cea5a6d1621249d46ef5b2cef915d8723cb9bfcce5d379d381e2219: Status 404 returned error can't find the container with id 8aee71c81cea5a6d1621249d46ef5b2cef915d8723cb9bfcce5d379d381e2219 Sep 16 13:13:29.597973 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:29.597939 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" event={"ID":"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1","Type":"ContainerStarted","Data":"8aee71c81cea5a6d1621249d46ef5b2cef915d8723cb9bfcce5d379d381e2219"} Sep 16 13:13:31.609989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:31.609930 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" event={"ID":"c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1","Type":"ContainerStarted","Data":"cb9d1d3139f921d4cc15184ba3557bab67ada0561588d0a5dd25dce8bfc7e420"} Sep 16 13:13:31.632487 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:31.632429 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bfz7b" podStartSLOduration=1.60085141 podStartE2EDuration="3.632411913s" totalImagesPullingTime="2.031560503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:31.632156257 +0000 UTC m=+296.751882803" watchObservedRunningTime="2026-09-16 13:13:31.632411913 +0000 UTC m=+296.752138462" Sep 16 13:13:32.290747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:32.290714 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:32.423343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:32.423309 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:34.020962 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.020926 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7"] Sep 16 13:13:34.024153 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.024136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.029036 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.029013 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:13:34.029189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.029111 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:13:34.029929 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.029914 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bjxjv\"" Sep 16 13:13:34.042263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.040251 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7"] Sep 16 13:13:34.048790 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.048765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.048889 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.048798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwf2\" (UniqueName: \"kubernetes.io/projected/517882df-7265-4036-bf7a-c8f2f782e22f-kube-api-access-bqwf2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.048931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.048911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.150256 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.150223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.150256 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.150261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.150477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.150295 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwf2\" (UniqueName: \"kubernetes.io/projected/517882df-7265-4036-bf7a-c8f2f782e22f-kube-api-access-bqwf2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.150630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.150610 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.150667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.150637 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.159426 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.159401 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwf2\" (UniqueName: \"kubernetes.io/projected/517882df-7265-4036-bf7a-c8f2f782e22f-kube-api-access-bqwf2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.265131 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.265095 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:34.333088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.333015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:34.609042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.608963 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7"] Sep 16 13:13:34.612651 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:13:34.612621 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517882df_7265_4036_bf7a_c8f2f782e22f.slice/crio-105eafc9837450fbf10e7fea49e35f7afe6f36edae6d345056c75c7557ecaee5 WatchSource:0}: Error finding container 105eafc9837450fbf10e7fea49e35f7afe6f36edae6d345056c75c7557ecaee5: Status 404 returned error can't find the container with id 105eafc9837450fbf10e7fea49e35f7afe6f36edae6d345056c75c7557ecaee5 Sep 16 13:13:34.623074 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:34.623047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" event={"ID":"517882df-7265-4036-bf7a-c8f2f782e22f","Type":"ContainerStarted","Data":"105eafc9837450fbf10e7fea49e35f7afe6f36edae6d345056c75c7557ecaee5"} Sep 16 13:13:35.308836 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:35.308812 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:13:35.309352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:35.309333 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:13:35.316077 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:35.316057 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 13:13:35.628929 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:35.628899 2581 generic.go:356] "Generic (PLEG): container finished" podID="517882df-7265-4036-bf7a-c8f2f782e22f" containerID="ceca63bef5a638be025a28507e2b266918b311d91d16c9eff0cdcd3b3d207148" exitCode=0 Sep 16 13:13:35.631124 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:35.628989 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" event={"ID":"517882df-7265-4036-bf7a-c8f2f782e22f","Type":"ContainerDied","Data":"ceca63bef5a638be025a28507e2b266918b311d91d16c9eff0cdcd3b3d207148"} Sep 16 13:13:36.635004 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:36.634934 2581 generic.go:356] "Generic (PLEG): container finished" podID="517882df-7265-4036-bf7a-c8f2f782e22f" containerID="5df277c1ffc543d7d05b0ab3334944232064fe559d29a7266ed3c018181f8422" exitCode=0 Sep 16 13:13:36.635350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:36.635027 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" event={"ID":"517882df-7265-4036-bf7a-c8f2f782e22f","Type":"ContainerDied","Data":"5df277c1ffc543d7d05b0ab3334944232064fe559d29a7266ed3c018181f8422"} Sep 16 13:13:37.640775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:37.640733 2581 generic.go:356] "Generic (PLEG): container finished" podID="517882df-7265-4036-bf7a-c8f2f782e22f" containerID="91afa16af6180b5f0a8a6ba125cda8a526e7605cce75c754f55d53f0d4057526" exitCode=0 Sep 16 13:13:37.641174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:37.640793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" event={"ID":"517882df-7265-4036-bf7a-c8f2f782e22f","Type":"ContainerDied","Data":"91afa16af6180b5f0a8a6ba125cda8a526e7605cce75c754f55d53f0d4057526"} Sep 16 13:13:38.784275 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.784247 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:38.885757 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.885729 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-util\") pod \"517882df-7265-4036-bf7a-c8f2f782e22f\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " Sep 16 13:13:38.885937 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.885794 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-bundle\") pod \"517882df-7265-4036-bf7a-c8f2f782e22f\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " Sep 16 13:13:38.885937 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.885906 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqwf2\" (UniqueName: \"kubernetes.io/projected/517882df-7265-4036-bf7a-c8f2f782e22f-kube-api-access-bqwf2\") pod \"517882df-7265-4036-bf7a-c8f2f782e22f\" (UID: \"517882df-7265-4036-bf7a-c8f2f782e22f\") " Sep 16 13:13:38.886521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.886492 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-bundle" (OuterVolumeSpecName: "bundle") pod "517882df-7265-4036-bf7a-c8f2f782e22f" (UID: "517882df-7265-4036-bf7a-c8f2f782e22f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:38.887924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.887897 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517882df-7265-4036-bf7a-c8f2f782e22f-kube-api-access-bqwf2" (OuterVolumeSpecName: "kube-api-access-bqwf2") pod "517882df-7265-4036-bf7a-c8f2f782e22f" (UID: "517882df-7265-4036-bf7a-c8f2f782e22f"). InnerVolumeSpecName "kube-api-access-bqwf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:13:38.893098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.893073 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-util" (OuterVolumeSpecName: "util") pod "517882df-7265-4036-bf7a-c8f2f782e22f" (UID: "517882df-7265-4036-bf7a-c8f2f782e22f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:38.986880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.986850 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqwf2\" (UniqueName: \"kubernetes.io/projected/517882df-7265-4036-bf7a-c8f2f782e22f-kube-api-access-bqwf2\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:38.986880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.986877 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:38.987042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:38.986887 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/517882df-7265-4036-bf7a-c8f2f782e22f-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:39.651026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:39.650943 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" event={"ID":"517882df-7265-4036-bf7a-c8f2f782e22f","Type":"ContainerDied","Data":"105eafc9837450fbf10e7fea49e35f7afe6f36edae6d345056c75c7557ecaee5"} Sep 16 13:13:39.651026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:39.650963 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft2wf7" Sep 16 13:13:39.651026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:39.650987 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105eafc9837450fbf10e7fea49e35f7afe6f36edae6d345056c75c7557ecaee5" Sep 16 13:13:46.702454 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:46.702420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:47.880400 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:47.880361 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:48.732070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732032 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92"] Sep 16 13:13:48.732441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732425 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="extract" Sep 16 13:13:48.732499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732444 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="extract" Sep 16 13:13:48.732499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732472 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="pull" Sep 16 13:13:48.732499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732480 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="pull" Sep 16 13:13:48.732499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732494 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="util" Sep 16 13:13:48.732623 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732502 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="util" Sep 16 13:13:48.732623 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.732566 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="517882df-7265-4036-bf7a-c8f2f782e22f" containerName="extract" Sep 16 13:13:48.735721 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.735701 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.740533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.740506 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:13:48.740648 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.740543 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bjxjv\"" Sep 16 13:13:48.748931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.748911 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:13:48.756441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.756418 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92"] Sep 16 13:13:48.859061 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.859031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.859237 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.859069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.859237 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.859102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflqh\" (UniqueName: \"kubernetes.io/projected/b6555362-7436-4454-95cf-f693c240917b-kube-api-access-mflqh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.959632 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.959600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.959982 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.959657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflqh\" (UniqueName: \"kubernetes.io/projected/b6555362-7436-4454-95cf-f693c240917b-kube-api-access-mflqh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.959982 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.959699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.959982 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.959974 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.960094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.960012 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:48.976404 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:48.976373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflqh\" (UniqueName: \"kubernetes.io/projected/b6555362-7436-4454-95cf-f693c240917b-kube-api-access-mflqh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:49.045374 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.045313 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:49.349164 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.349091 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92"] Sep 16 13:13:49.352666 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:13:49.352637 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6555362_7436_4454_95cf_f693c240917b.slice/crio-23633bb4dd603a070cb4c546dffe2810a8372a635f7c646ffe1bbaa78ad1fb4b WatchSource:0}: Error finding container 23633bb4dd603a070cb4c546dffe2810a8372a635f7c646ffe1bbaa78ad1fb4b: Status 404 returned error can't find the container with id 23633bb4dd603a070cb4c546dffe2810a8372a635f7c646ffe1bbaa78ad1fb4b Sep 16 13:13:49.659837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.659755 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:49.691384 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.691353 2581 generic.go:356] "Generic (PLEG): container finished" podID="b6555362-7436-4454-95cf-f693c240917b" containerID="ea39cbd0c3a9211499f496f959d7f6b41106a4f1e9a1e113e753322ef7107be8" exitCode=0 Sep 16 13:13:49.691521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.691442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" event={"ID":"b6555362-7436-4454-95cf-f693c240917b","Type":"ContainerDied","Data":"ea39cbd0c3a9211499f496f959d7f6b41106a4f1e9a1e113e753322ef7107be8"} Sep 16 13:13:49.691521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.691491 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" event={"ID":"b6555362-7436-4454-95cf-f693c240917b","Type":"ContainerStarted","Data":"23633bb4dd603a070cb4c546dffe2810a8372a635f7c646ffe1bbaa78ad1fb4b"} Sep 16 13:13:49.691892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.691878 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:13:49.783943 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:49.783906 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:50.265088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:50.265020 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:50.697787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:50.697701 2581 generic.go:356] "Generic (PLEG): container finished" podID="b6555362-7436-4454-95cf-f693c240917b" containerID="9a0627e20d031ef69c68ac96a478218f046de15164d49f6c7e87c4de6d0573f3" exitCode=0 Sep 16 13:13:50.697787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:50.697762 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" event={"ID":"b6555362-7436-4454-95cf-f693c240917b","Type":"ContainerDied","Data":"9a0627e20d031ef69c68ac96a478218f046de15164d49f6c7e87c4de6d0573f3"} Sep 16 13:13:51.173438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.173391 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:51.331888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.331856 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:51.365903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.365871 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr"] Sep 16 13:13:51.369090 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.369075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.372557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.372537 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 13:13:51.373800 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.373782 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 13:13:51.373800 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.373795 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 13:13:51.375431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.375366 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qxzqt\"" Sep 16 13:13:51.378006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.377989 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 13:13:51.406169 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.406143 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr"] Sep 16 13:13:51.478634 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.478609 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22ae90fe-7952-4ec2-9476-311dce8b45c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.478779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.478648 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22ae90fe-7952-4ec2-9476-311dce8b45c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.478779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.478670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862vz\" (UniqueName: \"kubernetes.io/projected/22ae90fe-7952-4ec2-9476-311dce8b45c5-kube-api-access-862vz\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.478779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.478743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ae90fe-7952-4ec2-9476-311dce8b45c5-tmp\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.478885 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.478785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22ae90fe-7952-4ec2-9476-311dce8b45c5-manifests\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.579744 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.579713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862vz\" (UniqueName: \"kubernetes.io/projected/22ae90fe-7952-4ec2-9476-311dce8b45c5-kube-api-access-862vz\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.579915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.579758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ae90fe-7952-4ec2-9476-311dce8b45c5-tmp\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.579915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.579784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22ae90fe-7952-4ec2-9476-311dce8b45c5-manifests\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.579915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.579905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22ae90fe-7952-4ec2-9476-311dce8b45c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.580125 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.579965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22ae90fe-7952-4ec2-9476-311dce8b45c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.580205 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.580175 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22ae90fe-7952-4ec2-9476-311dce8b45c5-manifests\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.582007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.581975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ae90fe-7952-4ec2-9476-311dce8b45c5-tmp\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.582271 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.582253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22ae90fe-7952-4ec2-9476-311dce8b45c5-webhook-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.582347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.582327 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22ae90fe-7952-4ec2-9476-311dce8b45c5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.593546 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.593524 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-862vz\" (UniqueName: \"kubernetes.io/projected/22ae90fe-7952-4ec2-9476-311dce8b45c5-kube-api-access-862vz\") pod \"opendatahub-operator-controller-manager-694ff5d95c-7x9xr\" (UID: \"22ae90fe-7952-4ec2-9476-311dce8b45c5\") " pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.671513 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.671467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:51.679273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.679248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:51.705589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.705523 2581 generic.go:356] "Generic (PLEG): container finished" podID="b6555362-7436-4454-95cf-f693c240917b" containerID="4c65474b555fd44b6cbc4eb9460f6381356f1799f345041c46096778e263ef59" exitCode=0 Sep 16 13:13:51.705705 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.705617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" event={"ID":"b6555362-7436-4454-95cf-f693c240917b","Type":"ContainerDied","Data":"4c65474b555fd44b6cbc4eb9460f6381356f1799f345041c46096778e263ef59"} Sep 16 13:13:51.787720 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.787683 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:51.957599 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.957563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:51.974494 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:51.974466 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr"] Sep 16 13:13:51.978641 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:13:51.978614 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ae90fe_7952_4ec2_9476_311dce8b45c5.slice/crio-70482ceaae4bbdd72371ed11907d5cfe8dde2cd6c868ac44fe07d8988e69b388 WatchSource:0}: Error finding container 70482ceaae4bbdd72371ed11907d5cfe8dde2cd6c868ac44fe07d8988e69b388: Status 404 returned error can't find the container with id 70482ceaae4bbdd72371ed11907d5cfe8dde2cd6c868ac44fe07d8988e69b388 Sep 16 13:13:52.077544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.077513 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:52.187414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.187376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:52.361158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.361065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:13:52.711344 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.711312 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" event={"ID":"22ae90fe-7952-4ec2-9476-311dce8b45c5","Type":"ContainerStarted","Data":"70482ceaae4bbdd72371ed11907d5cfe8dde2cd6c868ac44fe07d8988e69b388"} Sep 16 13:13:52.831556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.831531 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:52.890801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.890777 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-bundle\") pod \"b6555362-7436-4454-95cf-f693c240917b\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " Sep 16 13:13:52.890921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.890845 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-util\") pod \"b6555362-7436-4454-95cf-f693c240917b\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " Sep 16 13:13:52.890921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.890903 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mflqh\" (UniqueName: \"kubernetes.io/projected/b6555362-7436-4454-95cf-f693c240917b-kube-api-access-mflqh\") pod \"b6555362-7436-4454-95cf-f693c240917b\" (UID: \"b6555362-7436-4454-95cf-f693c240917b\") " Sep 16 13:13:52.891517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.891491 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-bundle" (OuterVolumeSpecName: "bundle") pod "b6555362-7436-4454-95cf-f693c240917b" (UID: "b6555362-7436-4454-95cf-f693c240917b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:52.892943 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.892922 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6555362-7436-4454-95cf-f693c240917b-kube-api-access-mflqh" (OuterVolumeSpecName: "kube-api-access-mflqh") pod "b6555362-7436-4454-95cf-f693c240917b" (UID: "b6555362-7436-4454-95cf-f693c240917b"). InnerVolumeSpecName "kube-api-access-mflqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:13:52.898996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.898974 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-util" (OuterVolumeSpecName: "util") pod "b6555362-7436-4454-95cf-f693c240917b" (UID: "b6555362-7436-4454-95cf-f693c240917b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:13:52.991849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.991775 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mflqh\" (UniqueName: \"kubernetes.io/projected/b6555362-7436-4454-95cf-f693c240917b-kube-api-access-mflqh\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:52.991849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.991800 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:52.991849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:52.991813 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6555362-7436-4454-95cf-f693c240917b-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:13:53.719177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:53.719140 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" event={"ID":"b6555362-7436-4454-95cf-f693c240917b","Type":"ContainerDied","Data":"23633bb4dd603a070cb4c546dffe2810a8372a635f7c646ffe1bbaa78ad1fb4b"} Sep 16 13:13:53.719678 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:53.719207 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kwj92" Sep 16 13:13:53.719678 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:53.719242 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23633bb4dd603a070cb4c546dffe2810a8372a635f7c646ffe1bbaa78ad1fb4b" Sep 16 13:13:55.730089 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:55.730054 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" event={"ID":"22ae90fe-7952-4ec2-9476-311dce8b45c5","Type":"ContainerStarted","Data":"ad7642f62dd4650f5ca66737da54caff4293071795fc48dee8de749f1be75d5e"} Sep 16 13:13:56.735441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:56.735409 2581 generic.go:356] "Generic (PLEG): container finished" podID="22ae90fe-7952-4ec2-9476-311dce8b45c5" containerID="ad7642f62dd4650f5ca66737da54caff4293071795fc48dee8de749f1be75d5e" exitCode=0 Sep 16 13:13:56.735801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:56.735454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" event={"ID":"22ae90fe-7952-4ec2-9476-311dce8b45c5","Type":"ContainerDied","Data":"ad7642f62dd4650f5ca66737da54caff4293071795fc48dee8de749f1be75d5e"} Sep 16 13:13:57.741683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:57.741651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" event={"ID":"22ae90fe-7952-4ec2-9476-311dce8b45c5","Type":"ContainerStarted","Data":"c67bde8ab2d7461ad5c4f571839a5f5ef252991cb5154b4307010b2e8ff05281"} Sep 16 13:13:57.742067 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:57.741784 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:13:57.768702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:13:57.768655 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" podStartSLOduration=2.126019901 podStartE2EDuration="6.768639392s" totalImagesPullingTime="3.631161617s" totalInitContainerRuntime="1.011457874s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:57.766674508 +0000 UTC m=+322.886401075" watchObservedRunningTime="2026-09-16 13:13:57.768639392 +0000 UTC m=+322.888365942" Sep 16 13:14:00.567325 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:00.567273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:00.646223 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:00.646176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:08.748058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:08.748024 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694ff5d95c-7x9xr" Sep 16 13:14:10.901499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:10.901454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:11.076251 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.076210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:11.386984 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.386947 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:11.423147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423111 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6"] Sep 16 13:14:11.423537 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423522 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="extract" Sep 16 13:14:11.423587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423541 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="extract" Sep 16 13:14:11.423587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423555 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="pull" Sep 16 13:14:11.423587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423563 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="pull" Sep 16 13:14:11.423681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423594 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="util" Sep 16 13:14:11.423681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423603 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="util" Sep 16 13:14:11.423681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.423671 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6555362-7436-4454-95cf-f693c240917b" containerName="extract" Sep 16 13:14:11.436252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.436222 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.441751 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.441720 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 13:14:11.444842 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.444814 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6"] Sep 16 13:14:11.445232 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.445210 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 13:14:11.445377 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.445360 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 13:14:11.447677 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.447661 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hql2q\"" Sep 16 13:14:11.454025 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.454009 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 13:14:11.535647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.535608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2585c164-4944-4ccc-98a7-750dcec12145-tls-certs\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.535815 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.535713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmn5\" (UniqueName: \"kubernetes.io/projected/2585c164-4944-4ccc-98a7-750dcec12145-kube-api-access-6xmn5\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.535815 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.535769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2585c164-4944-4ccc-98a7-750dcec12145-tmp\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.636633 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.636595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2585c164-4944-4ccc-98a7-750dcec12145-tmp\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.636798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.636641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2585c164-4944-4ccc-98a7-750dcec12145-tls-certs\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.636798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.636687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmn5\" (UniqueName: \"kubernetes.io/projected/2585c164-4944-4ccc-98a7-750dcec12145-kube-api-access-6xmn5\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.636916 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:14:11.636812 2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 16 13:14:11.636916 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:14:11.636900 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2585c164-4944-4ccc-98a7-750dcec12145-tls-certs podName:2585c164-4944-4ccc-98a7-750dcec12145 nodeName:}" failed. No retries permitted until 2026-09-16 13:14:12.136877182 +0000 UTC m=+337.256603723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2585c164-4944-4ccc-98a7-750dcec12145-tls-certs") pod "kube-auth-proxy-6445dbdc5b-gkbm6" (UID: "2585c164-4944-4ccc-98a7-750dcec12145") : secret "kube-auth-proxy-tls" not found Sep 16 13:14:11.638929 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.638873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2585c164-4944-4ccc-98a7-750dcec12145-tmp\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.646121 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.646103 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmn5\" (UniqueName: \"kubernetes.io/projected/2585c164-4944-4ccc-98a7-750dcec12145-kube-api-access-6xmn5\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:11.997410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:11.997368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:12.140302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:12.140249 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2585c164-4944-4ccc-98a7-750dcec12145-tls-certs\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:12.142871 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:12.142837 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2585c164-4944-4ccc-98a7-750dcec12145-tls-certs\") pod \"kube-auth-proxy-6445dbdc5b-gkbm6\" (UID: \"2585c164-4944-4ccc-98a7-750dcec12145\") " pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:12.349073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:12.348990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" Sep 16 13:14:12.625356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:12.625322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6"] Sep 16 13:14:12.627777 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:14:12.627751 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2585c164_4944_4ccc_98a7_750dcec12145.slice/crio-6c2f54749121b90e1768903cce1cf133ab5055968cf5467ef21cdad832ed83dc WatchSource:0}: Error finding container 6c2f54749121b90e1768903cce1cf133ab5055968cf5467ef21cdad832ed83dc: Status 404 returned error can't find the container with id 6c2f54749121b90e1768903cce1cf133ab5055968cf5467ef21cdad832ed83dc Sep 16 13:14:12.814114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:12.814079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" event={"ID":"2585c164-4944-4ccc-98a7-750dcec12145","Type":"ContainerStarted","Data":"6c2f54749121b90e1768903cce1cf133ab5055968cf5467ef21cdad832ed83dc"} Sep 16 13:14:13.043733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:13.043688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:13.136643 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:13.136610 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:14.098800 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:14.098689 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:15.831269 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:15.831234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" event={"ID":"2585c164-4944-4ccc-98a7-750dcec12145","Type":"ContainerStarted","Data":"6e458a2bcd8ee27f2e62ef708848669ac667995b3ce3d87aac859f0c016b9050"} Sep 16 13:14:15.848885 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:15.848839 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6445dbdc5b-gkbm6" podStartSLOduration=1.862846137 podStartE2EDuration="4.848823651s" totalImagesPullingTime="2.985977514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:15.846747498 +0000 UTC m=+340.966474047" watchObservedRunningTime="2026-09-16 13:14:15.848823651 +0000 UTC m=+340.968550225" Sep 16 13:14:18.281513 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.281471 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:18.383181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.383144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:18.497207 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.497164 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:18.642194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.642120 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff"] Sep 16 13:14:18.645465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.645450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.649082 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.649062 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 13:14:18.649213 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.649110 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2qvlv\"" Sep 16 13:14:18.660472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.660450 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff"] Sep 16 13:14:18.793998 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.793969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794159 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794159 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794159 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7790b7b1-ee07-4883-a478-3ed9c23a986e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794159 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgb6\" (UniqueName: \"kubernetes.io/projected/7790b7b1-ee07-4883-a478-3ed9c23a986e-kube-api-access-zxgb6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.794355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.794297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.872515 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.872488 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:18.895529 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895529 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895529 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7790b7b1-ee07-4883-a478-3ed9c23a986e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgb6\" (UniqueName: \"kubernetes.io/projected/7790b7b1-ee07-4883-a478-3ed9c23a986e-kube-api-access-zxgb6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895904 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.895988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.895979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.896229 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.896029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.896335 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.896312 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.896479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.896460 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7790b7b1-ee07-4883-a478-3ed9c23a986e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.897862 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.897842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.898019 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.898000 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.903380 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.903359 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7790b7b1-ee07-4883-a478-3ed9c23a986e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.903486 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.903433 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgb6\" (UniqueName: \"kubernetes.io/projected/7790b7b1-ee07-4883-a478-3ed9c23a986e-kube-api-access-zxgb6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff\" (UID: \"7790b7b1-ee07-4883-a478-3ed9c23a986e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:18.944410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.944384 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:18.955982 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:18.955958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:19.231368 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:19.231343 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff"] Sep 16 13:14:19.233562 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:14:19.233537 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7790b7b1_ee07_4883_a478_3ed9c23a986e.slice/crio-61b4a5a10fd9527a8eb4f530b28d58acda68fb03d876d72c15672179b534bb1c WatchSource:0}: Error finding container 61b4a5a10fd9527a8eb4f530b28d58acda68fb03d876d72c15672179b534bb1c: Status 404 returned error can't find the container with id 61b4a5a10fd9527a8eb4f530b28d58acda68fb03d876d72c15672179b534bb1c Sep 16 13:14:19.851811 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:19.851742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" event={"ID":"7790b7b1-ee07-4883-a478-3ed9c23a986e","Type":"ContainerStarted","Data":"61b4a5a10fd9527a8eb4f530b28d58acda68fb03d876d72c15672179b534bb1c"} Sep 16 13:14:21.656369 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.656331 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:14:21.656604 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.656410 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:14:21.656604 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.656439 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:14:21.863310 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.863251 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" event={"ID":"7790b7b1-ee07-4883-a478-3ed9c23a986e","Type":"ContainerStarted","Data":"e2a39323c4a6a4dcbc6fd82afce83959966fed53fa0b34e810c3c472930c556a"} Sep 16 13:14:21.891239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.891209 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sqljq"] Sep 16 13:14:21.894490 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.894445 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" podStartSLOduration=1.474146411 podStartE2EDuration="3.894425608s" totalImagesPullingTime="2.420279197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:21.893632817 +0000 UTC m=+347.013359386" watchObservedRunningTime="2026-09-16 13:14:21.894425608 +0000 UTC m=+347.014152159" Sep 16 13:14:21.894636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.894557 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:21.901614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.901457 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 16 13:14:21.901779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.901703 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jmfzm\"" Sep 16 13:14:21.908450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.908379 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sqljq"] Sep 16 13:14:21.912185 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.912154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:21.956086 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.956058 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:21.983953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:21.983920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:22.022890 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.022868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-cert\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.023040 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.023013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgkk\" (UniqueName: \"kubernetes.io/projected/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-kube-api-access-8vgkk\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.124346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.124311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-cert\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.124534 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.124378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgkk\" (UniqueName: \"kubernetes.io/projected/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-kube-api-access-8vgkk\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.124534 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:14:22.124477 2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 16 13:14:22.124646 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:14:22.124572 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-cert podName:cd31f8ec-13d8-4a5f-a96c-678b72e4d82b nodeName:}" failed. No retries permitted until 2026-09-16 13:14:22.624549001 +0000 UTC m=+347.744275533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-cert") pod "odh-model-controller-7998c67dc4-sqljq" (UID: "cd31f8ec-13d8-4a5f-a96c-678b72e4d82b") : secret "odh-model-controller-webhook-cert" not found Sep 16 13:14:22.134752 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.134717 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgkk\" (UniqueName: \"kubernetes.io/projected/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-kube-api-access-8vgkk\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.350200 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.350169 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:22.409078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.409044 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kpbdv"] Sep 16 13:14:22.413848 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.413830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.416223 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.416202 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 16 13:14:22.416383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.416358 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-99mpx\"" Sep 16 13:14:22.421876 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.421850 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kpbdv"] Sep 16 13:14:22.526859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.526832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1439ca8-6576-4e08-8093-cea64ff568cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-kpbdv\" (UID: \"c1439ca8-6576-4e08-8093-cea64ff568cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.527021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.526866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqh6b\" (UniqueName: \"kubernetes.io/projected/c1439ca8-6576-4e08-8093-cea64ff568cb-kube-api-access-fqh6b\") pod \"model-serving-api-54d89bd5fc-kpbdv\" (UID: \"c1439ca8-6576-4e08-8093-cea64ff568cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.569838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.569811 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:22.627642 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.627567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-cert\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.627642 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.627600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1439ca8-6576-4e08-8093-cea64ff568cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-kpbdv\" (UID: \"c1439ca8-6576-4e08-8093-cea64ff568cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.627642 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.627622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqh6b\" (UniqueName: \"kubernetes.io/projected/c1439ca8-6576-4e08-8093-cea64ff568cb-kube-api-access-fqh6b\") pod \"model-serving-api-54d89bd5fc-kpbdv\" (UID: \"c1439ca8-6576-4e08-8093-cea64ff568cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.630115 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.630086 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1439ca8-6576-4e08-8093-cea64ff568cb-tls-certs\") pod \"model-serving-api-54d89bd5fc-kpbdv\" (UID: \"c1439ca8-6576-4e08-8093-cea64ff568cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.630216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.630086 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd31f8ec-13d8-4a5f-a96c-678b72e4d82b-cert\") pod \"odh-model-controller-7998c67dc4-sqljq\" (UID: \"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b\") " pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.637901 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.637877 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqh6b\" (UniqueName: \"kubernetes.io/projected/c1439ca8-6576-4e08-8093-cea64ff568cb-kube-api-access-fqh6b\") pod \"model-serving-api-54d89bd5fc-kpbdv\" (UID: \"c1439ca8-6576-4e08-8093-cea64ff568cb\") " pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.670108 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.670078 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:22.725515 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.725495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:22.812201 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.812173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:22.961198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.961172 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:22.993609 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:22.993582 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kpbdv"] Sep 16 13:14:22.996982 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:14:22.996953 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1439ca8_6576_4e08_8093_cea64ff568cb.slice/crio-94b46de2d947e80dd6e4ab68fd84ab71346e0c3007478193bdecb7c898715493 WatchSource:0}: Error finding container 94b46de2d947e80dd6e4ab68fd84ab71346e0c3007478193bdecb7c898715493: Status 404 returned error can't find the container with id 94b46de2d947e80dd6e4ab68fd84ab71346e0c3007478193bdecb7c898715493 Sep 16 13:14:23.082883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:23.082861 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sqljq"] Sep 16 13:14:23.084981 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:14:23.084960 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd31f8ec_13d8_4a5f_a96c_678b72e4d82b.slice/crio-3e61bd97a19a8e335bfcf806da229a5e0c0b27f04037eb517abf4d19ba9afda6 WatchSource:0}: Error finding container 3e61bd97a19a8e335bfcf806da229a5e0c0b27f04037eb517abf4d19ba9afda6: Status 404 returned error can't find the container with id 3e61bd97a19a8e335bfcf806da229a5e0c0b27f04037eb517abf4d19ba9afda6 Sep 16 13:14:23.875247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:23.875212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" event={"ID":"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b","Type":"ContainerStarted","Data":"3e61bd97a19a8e335bfcf806da229a5e0c0b27f04037eb517abf4d19ba9afda6"} Sep 16 13:14:23.876437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:23.876412 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" event={"ID":"c1439ca8-6576-4e08-8093-cea64ff568cb","Type":"ContainerStarted","Data":"94b46de2d947e80dd6e4ab68fd84ab71346e0c3007478193bdecb7c898715493"} Sep 16 13:14:23.876584 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:23.876563 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:23.877546 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:23.877528 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff" Sep 16 13:14:25.887812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:25.887768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" event={"ID":"c1439ca8-6576-4e08-8093-cea64ff568cb","Type":"ContainerStarted","Data":"c75ba931c3a99ffec462eef8111d767909b83ea2316c080df6fb4f4bb0861efe"} Sep 16 13:14:25.888322 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:25.888123 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:25.905583 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:25.905535 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" podStartSLOduration=2.00869988 podStartE2EDuration="3.905517829s" totalImagesPullingTime="1.896817949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:25.903171602 +0000 UTC m=+351.022898158" watchObservedRunningTime="2026-09-16 13:14:25.905517829 +0000 UTC m=+351.025244377" Sep 16 13:14:26.895102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:26.895064 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" event={"ID":"cd31f8ec-13d8-4a5f-a96c-678b72e4d82b","Type":"ContainerStarted","Data":"220468812cfcd369184aa9616ff4dddde67f01212a51ee1b77443a95454886d0"} Sep 16 13:14:26.912563 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:26.912515 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" podStartSLOduration=2.643346567 podStartE2EDuration="5.912498034s" totalImagesPullingTime="3.269151467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:26.909570582 +0000 UTC m=+352.029297150" watchObservedRunningTime="2026-09-16 13:14:26.912498034 +0000 UTC m=+352.032224582" Sep 16 13:14:27.899537 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:27.899462 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:14:28.665426 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.665377 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df"] Sep 16 13:14:28.668400 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.668384 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.670798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.670777 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 13:14:28.670798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.670785 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 13:14:28.670984 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.670804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2znts\"" Sep 16 13:14:28.670984 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.670779 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.674946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.674994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.675021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88ln\" (UniqueName: \"kubernetes.io/projected/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-kube-api-access-m88ln\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.675078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.675142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.675234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.675433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.675322 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.684996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.684973 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df"] Sep 16 13:14:28.775927 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.775890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.775927 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.775926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88ln\" (UniqueName: \"kubernetes.io/projected/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-kube-api-access-m88ln\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.776139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.775952 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.776139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.775994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.776139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.776025 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.776139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.776065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.776139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.776127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.776918 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.776885 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.778511 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.778478 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.778618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.778537 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.778618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.778550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.778618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.778584 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.798065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.798036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88ln\" (UniqueName: \"kubernetes.io/projected/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-kube-api-access-m88ln\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.798065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.798062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-l45df\" (UID: \"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:28.977581 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:28.977552 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:29.265843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:29.265766 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df"] Sep 16 13:14:29.268979 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:14:29.268953 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec86c9b_8f0a_4cb6_9964_9eebe4fc0075.slice/crio-215829aebf923b693d5cec93e4694b9cec571c086c1f346d43891d6e8199559a WatchSource:0}: Error finding container 215829aebf923b693d5cec93e4694b9cec571c086c1f346d43891d6e8199559a: Status 404 returned error can't find the container with id 215829aebf923b693d5cec93e4694b9cec571c086c1f346d43891d6e8199559a Sep 16 13:14:29.909883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:29.909793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" event={"ID":"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075","Type":"ContainerStarted","Data":"215829aebf923b693d5cec93e4694b9cec571c086c1f346d43891d6e8199559a"} Sep 16 13:14:31.409150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.409110 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:31.506368 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.506276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:31.640760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.640685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:31.820007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.819971 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:31.847416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.847376 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:14:31.847519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.847442 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:14:31.922781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.922738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" event={"ID":"6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075","Type":"ContainerStarted","Data":"f093c8fdfe95f8788ff704c1ccf9797175ec873409cf2672cd8e336fa7af4a4b"} Sep 16 13:14:31.922940 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.922813 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:31.944638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:31.944576 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" podStartSLOduration=1.368632463 podStartE2EDuration="3.944559739s" totalImagesPullingTime="2.575927276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:14:31.943262495 +0000 UTC m=+357.062989073" watchObservedRunningTime="2026-09-16 13:14:31.944559739 +0000 UTC m=+357.064286288" Sep 16 13:14:32.668276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:32.668246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:32.776916 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:32.776888 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:32.868933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:32.868899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:14:32.929116 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:32.929045 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-l45df" Sep 16 13:14:37.904051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:37.904018 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-kpbdv" Sep 16 13:14:38.906761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:14:38.906733 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sqljq" Sep 16 13:15:18.000024 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:17.999989 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dbcf769b9-spqgv"] Sep 16 13:15:18.008330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.008303 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.017219 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.017187 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbcf769b9-spqgv"] Sep 16 13:15:18.058198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.058167 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:18.091181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-console-config\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.091343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-trusted-ca-bundle\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.091343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-service-ca\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.091343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a310f581-0a70-434f-a942-7aae07764397-console-serving-cert\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.091343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a310f581-0a70-434f-a942-7aae07764397-console-oauth-config\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.091484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwmf\" (UniqueName: \"kubernetes.io/projected/a310f581-0a70-434f-a942-7aae07764397-kube-api-access-ftwmf\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.091484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.091400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-oauth-serving-cert\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.127670 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.127645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:18.192463 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-console-config\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.192638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-trusted-ca-bundle\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.192638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-service-ca\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.192638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a310f581-0a70-434f-a942-7aae07764397-console-serving-cert\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.192638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a310f581-0a70-434f-a942-7aae07764397-console-oauth-config\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.192638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwmf\" (UniqueName: \"kubernetes.io/projected/a310f581-0a70-434f-a942-7aae07764397-kube-api-access-ftwmf\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.192638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.192625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-oauth-serving-cert\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.193271 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.193245 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-service-ca\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.193411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.193254 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-console-config\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.193411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.193311 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-oauth-serving-cert\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.193645 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.193622 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310f581-0a70-434f-a942-7aae07764397-trusted-ca-bundle\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.195036 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.195014 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a310f581-0a70-434f-a942-7aae07764397-console-oauth-config\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.195630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.195611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a310f581-0a70-434f-a942-7aae07764397-console-serving-cert\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.200448 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.200427 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwmf\" (UniqueName: \"kubernetes.io/projected/a310f581-0a70-434f-a942-7aae07764397-kube-api-access-ftwmf\") pod \"console-7dbcf769b9-spqgv\" (UID: \"a310f581-0a70-434f-a942-7aae07764397\") " pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.318482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.318401 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:18.594093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:18.594063 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbcf769b9-spqgv"] Sep 16 13:15:18.596675 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:15:18.596649 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda310f581_0a70_434f_a942_7aae07764397.slice/crio-c92eb944a8e8f7796c5f0ecd14fb85e5a3a4f8957e51c3ba8427a1a04698f598 WatchSource:0}: Error finding container c92eb944a8e8f7796c5f0ecd14fb85e5a3a4f8957e51c3ba8427a1a04698f598: Status 404 returned error can't find the container with id c92eb944a8e8f7796c5f0ecd14fb85e5a3a4f8957e51c3ba8427a1a04698f598 Sep 16 13:15:19.135977 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:19.135942 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbcf769b9-spqgv" event={"ID":"a310f581-0a70-434f-a942-7aae07764397","Type":"ContainerStarted","Data":"c9a91e1a98252e158e8b113589b910afce36827bfda33f28a2b2519a8c75df46"} Sep 16 13:15:19.136498 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:19.135988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbcf769b9-spqgv" event={"ID":"a310f581-0a70-434f-a942-7aae07764397","Type":"ContainerStarted","Data":"c92eb944a8e8f7796c5f0ecd14fb85e5a3a4f8957e51c3ba8427a1a04698f598"} Sep 16 13:15:19.155557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:19.155505 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dbcf769b9-spqgv" podStartSLOduration=2.155488693 podStartE2EDuration="2.155488693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:19.154584663 +0000 UTC m=+404.274311225" watchObservedRunningTime="2026-09-16 13:15:19.155488693 +0000 UTC m=+404.275215243" Sep 16 13:15:23.315267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:23.315213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:23.436728 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:23.436696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:27.278781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.278739 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc"] Sep 16 13:15:27.282630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.282613 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.284933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.284912 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:15:27.285705 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.285689 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:15:27.285767 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.285692 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bjxjv\"" Sep 16 13:15:27.292007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.291985 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc"] Sep 16 13:15:27.374177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.374143 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc"] Sep 16 13:15:27.377688 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.377671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.387652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.387630 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc"] Sep 16 13:15:27.466859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.466816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.466859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.466854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.467049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.466891 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.467049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.466984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.467049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.467021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728ff\" (UniqueName: \"kubernetes.io/projected/79ae829d-6052-4803-8f7a-f8baa01264ce-kube-api-access-728ff\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.467147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.467057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvp2l\" (UniqueName: \"kubernetes.io/projected/76a6054b-fbcc-4d42-abd8-687352722344-kube-api-access-mvp2l\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.475804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.475771 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c"] Sep 16 13:15:27.479423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.479403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.487961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.487939 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c"] Sep 16 13:15:27.567700 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvp2l\" (UniqueName: \"kubernetes.io/projected/76a6054b-fbcc-4d42-abd8-687352722344-kube-api-access-mvp2l\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.567700 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.567906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.567906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.567906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/dd449cf6-edc7-4ffd-997c-d0c1146b883a-kube-api-access-gtqvz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.567906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.568051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.568051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.567957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.568170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.568071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728ff\" (UniqueName: \"kubernetes.io/projected/79ae829d-6052-4803-8f7a-f8baa01264ce-kube-api-access-728ff\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.568170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.568136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.568234 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.568182 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.568234 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.568185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.568234 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.568208 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.579236 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.579201 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65"] Sep 16 13:15:27.581655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.581630 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvp2l\" (UniqueName: \"kubernetes.io/projected/76a6054b-fbcc-4d42-abd8-687352722344-kube-api-access-mvp2l\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.581655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.581640 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-728ff\" (UniqueName: \"kubernetes.io/projected/79ae829d-6052-4803-8f7a-f8baa01264ce-kube-api-access-728ff\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.583000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.582984 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.592032 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.592009 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65"] Sep 16 13:15:27.592127 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.592097 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:27.668930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.668902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.669069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.668943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/dd449cf6-edc7-4ffd-997c-d0c1146b883a-kube-api-access-gtqvz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.669069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.668976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.669316 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.669276 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.669397 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.669298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.677447 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.677427 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/dd449cf6-edc7-4ffd-997c-d0c1146b883a-kube-api-access-gtqvz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.688406 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.688380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:27.769718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.769688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.769858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.769737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.769858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.769765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxg6z\" (UniqueName: \"kubernetes.io/projected/b8df45d3-81fa-46bd-8eb4-4464949235de-kube-api-access-cxg6z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.788731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.788704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:27.871135 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.871111 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc"] Sep 16 13:15:27.871233 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.871210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.871308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.871256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.871308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.871278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxg6z\" (UniqueName: \"kubernetes.io/projected/b8df45d3-81fa-46bd-8eb4-4464949235de-kube-api-access-cxg6z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.871617 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.871599 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.871692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.871635 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.873114 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:15:27.873095 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae829d_6052_4803_8f7a_f8baa01264ce.slice/crio-e8887aeebf8425316bb2cfe9fc82c041055c83ba7584f1e4ffd983be23ca4b6b WatchSource:0}: Error finding container e8887aeebf8425316bb2cfe9fc82c041055c83ba7584f1e4ffd983be23ca4b6b: Status 404 returned error can't find the container with id e8887aeebf8425316bb2cfe9fc82c041055c83ba7584f1e4ffd983be23ca4b6b Sep 16 13:15:27.878887 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.878870 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxg6z\" (UniqueName: \"kubernetes.io/projected/b8df45d3-81fa-46bd-8eb4-4464949235de-kube-api-access-cxg6z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.903162 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.903105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:27.963546 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:27.963515 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc"] Sep 16 13:15:27.969672 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:15:27.969638 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a6054b_fbcc_4d42_abd8_687352722344.slice/crio-a46e36f6521168ed9f8a271b44284ff250e5e1d2f7ba9814d5be763a21b33df9 WatchSource:0}: Error finding container a46e36f6521168ed9f8a271b44284ff250e5e1d2f7ba9814d5be763a21b33df9: Status 404 returned error can't find the container with id a46e36f6521168ed9f8a271b44284ff250e5e1d2f7ba9814d5be763a21b33df9 Sep 16 13:15:28.060622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.060559 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c"] Sep 16 13:15:28.062635 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:15:28.062611 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd449cf6_edc7_4ffd_997c_d0c1146b883a.slice/crio-ddab2654e7546962ec27aa29eb4e39e7eb8e5d8872d0c254505324aea6e929fc WatchSource:0}: Error finding container ddab2654e7546962ec27aa29eb4e39e7eb8e5d8872d0c254505324aea6e929fc: Status 404 returned error can't find the container with id ddab2654e7546962ec27aa29eb4e39e7eb8e5d8872d0c254505324aea6e929fc Sep 16 13:15:28.179814 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.179506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" event={"ID":"dd449cf6-edc7-4ffd-997c-d0c1146b883a","Type":"ContainerStarted","Data":"1d0d95b130adbfa87942e2e291b20bfc11f0de797b7740d4c2d15ea5f38b1e75"} Sep 16 13:15:28.180166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.180131 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" event={"ID":"dd449cf6-edc7-4ffd-997c-d0c1146b883a","Type":"ContainerStarted","Data":"ddab2654e7546962ec27aa29eb4e39e7eb8e5d8872d0c254505324aea6e929fc"} Sep 16 13:15:28.181729 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.181704 2581 generic.go:356] "Generic (PLEG): container finished" podID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerID="cf18f6c00d9e7763fb03b11e51c3a10d8b1bd48c625db215523964d6dd17bce3" exitCode=0 Sep 16 13:15:28.181831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.181784 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" event={"ID":"79ae829d-6052-4803-8f7a-f8baa01264ce","Type":"ContainerDied","Data":"cf18f6c00d9e7763fb03b11e51c3a10d8b1bd48c625db215523964d6dd17bce3"} Sep 16 13:15:28.181894 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.181831 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" event={"ID":"79ae829d-6052-4803-8f7a-f8baa01264ce","Type":"ContainerStarted","Data":"e8887aeebf8425316bb2cfe9fc82c041055c83ba7584f1e4ffd983be23ca4b6b"} Sep 16 13:15:28.181947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.181902 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65"] Sep 16 13:15:28.183664 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.183641 2581 generic.go:356] "Generic (PLEG): container finished" podID="76a6054b-fbcc-4d42-abd8-687352722344" containerID="b9ff589e97aa72a8c959a51948e3ebbf95894e3316e26a47e2fdd30f845ce4c3" exitCode=0 Sep 16 13:15:28.183748 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.183715 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" event={"ID":"76a6054b-fbcc-4d42-abd8-687352722344","Type":"ContainerDied","Data":"b9ff589e97aa72a8c959a51948e3ebbf95894e3316e26a47e2fdd30f845ce4c3"} Sep 16 13:15:28.183791 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.183757 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" event={"ID":"76a6054b-fbcc-4d42-abd8-687352722344","Type":"ContainerStarted","Data":"a46e36f6521168ed9f8a271b44284ff250e5e1d2f7ba9814d5be763a21b33df9"} Sep 16 13:15:28.229111 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:15:28.229083 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8df45d3_81fa_46bd_8eb4_4464949235de.slice/crio-380e2ed067288db3d6a71cd67c30be3b46c661ab7efea4df219ea9e4c93af754 WatchSource:0}: Error finding container 380e2ed067288db3d6a71cd67c30be3b46c661ab7efea4df219ea9e4c93af754: Status 404 returned error can't find the container with id 380e2ed067288db3d6a71cd67c30be3b46c661ab7efea4df219ea9e4c93af754 Sep 16 13:15:28.319143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.319119 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:28.319516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.319412 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:28.323931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:28.323913 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:29.190194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.190162 2581 generic.go:356] "Generic (PLEG): container finished" podID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerID="1d0d95b130adbfa87942e2e291b20bfc11f0de797b7740d4c2d15ea5f38b1e75" exitCode=0 Sep 16 13:15:29.190341 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.190237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" event={"ID":"dd449cf6-edc7-4ffd-997c-d0c1146b883a","Type":"ContainerDied","Data":"1d0d95b130adbfa87942e2e291b20bfc11f0de797b7740d4c2d15ea5f38b1e75"} Sep 16 13:15:29.192738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.192714 2581 generic.go:356] "Generic (PLEG): container finished" podID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerID="19f69087af55ba22d5ae242efcc1c85d7e110f36ac1f0e90dfeb6608bbbc6f58" exitCode=0 Sep 16 13:15:29.192823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.192796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" event={"ID":"79ae829d-6052-4803-8f7a-f8baa01264ce","Type":"ContainerDied","Data":"19f69087af55ba22d5ae242efcc1c85d7e110f36ac1f0e90dfeb6608bbbc6f58"} Sep 16 13:15:29.194626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.194595 2581 generic.go:356] "Generic (PLEG): container finished" podID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerID="61b108f4acf5147e5ed6a309522968a7c4128ad936e36afba5ffbfa13d6018a7" exitCode=0 Sep 16 13:15:29.194724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.194700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" event={"ID":"b8df45d3-81fa-46bd-8eb4-4464949235de","Type":"ContainerDied","Data":"61b108f4acf5147e5ed6a309522968a7c4128ad936e36afba5ffbfa13d6018a7"} Sep 16 13:15:29.194904 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.194748 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" event={"ID":"b8df45d3-81fa-46bd-8eb4-4464949235de","Type":"ContainerStarted","Data":"380e2ed067288db3d6a71cd67c30be3b46c661ab7efea4df219ea9e4c93af754"} Sep 16 13:15:29.196687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.196663 2581 generic.go:356] "Generic (PLEG): container finished" podID="76a6054b-fbcc-4d42-abd8-687352722344" containerID="32a759274fa49f6e470f2d3b93d2ea3ef4b75d81b9507f6a6bec4699976f6402" exitCode=0 Sep 16 13:15:29.196760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.196723 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" event={"ID":"76a6054b-fbcc-4d42-abd8-687352722344","Type":"ContainerDied","Data":"32a759274fa49f6e470f2d3b93d2ea3ef4b75d81b9507f6a6bec4699976f6402"} Sep 16 13:15:29.201760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.201741 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dbcf769b9-spqgv" Sep 16 13:15:29.320834 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.320742 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf5d44cb9-tphb5"] Sep 16 13:15:29.436079 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.436052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:29.536523 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.536492 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:29.600677 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:29.600603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:30.204175 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:30.204146 2581 generic.go:356] "Generic (PLEG): container finished" podID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerID="19bd7c1cc02026f17da49c6c0f68b13d662bcf78e675f9865667bd223a0b3d28" exitCode=0 Sep 16 13:15:30.204307 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:30.204232 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" event={"ID":"dd449cf6-edc7-4ffd-997c-d0c1146b883a","Type":"ContainerDied","Data":"19bd7c1cc02026f17da49c6c0f68b13d662bcf78e675f9865667bd223a0b3d28"} Sep 16 13:15:30.206765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:30.206746 2581 generic.go:356] "Generic (PLEG): container finished" podID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerID="dd0e419f7f4cd66cfc8e5067376b1d3bbb430684dde5401e61b0fa812539ba95" exitCode=0 Sep 16 13:15:30.206860 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:30.206835 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" event={"ID":"79ae829d-6052-4803-8f7a-f8baa01264ce","Type":"ContainerDied","Data":"dd0e419f7f4cd66cfc8e5067376b1d3bbb430684dde5401e61b0fa812539ba95"} Sep 16 13:15:30.209219 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:30.209197 2581 generic.go:356] "Generic (PLEG): container finished" podID="76a6054b-fbcc-4d42-abd8-687352722344" containerID="0b7a3c7885f8da551503df207c3ff9e2932be084f1a67e1b234b4324c5126f2c" exitCode=0 Sep 16 13:15:30.209328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:30.209260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" event={"ID":"76a6054b-fbcc-4d42-abd8-687352722344","Type":"ContainerDied","Data":"0b7a3c7885f8da551503df207c3ff9e2932be084f1a67e1b234b4324c5126f2c"} Sep 16 13:15:31.216017 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.215982 2581 generic.go:356] "Generic (PLEG): container finished" podID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerID="0a5f2029f00a78ad642ebb53486e93cb4b08504b33c1d3505c3c054374a73189" exitCode=0 Sep 16 13:15:31.216392 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.216057 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" event={"ID":"b8df45d3-81fa-46bd-8eb4-4464949235de","Type":"ContainerDied","Data":"0a5f2029f00a78ad642ebb53486e93cb4b08504b33c1d3505c3c054374a73189"} Sep 16 13:15:31.218479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.218455 2581 generic.go:356] "Generic (PLEG): container finished" podID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerID="b39ae1d0e41d651bb047a98a7fa822f907fd0e0143668b9c7ae92ddcb471dee8" exitCode=0 Sep 16 13:15:31.218562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.218506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" event={"ID":"dd449cf6-edc7-4ffd-997c-d0c1146b883a","Type":"ContainerDied","Data":"b39ae1d0e41d651bb047a98a7fa822f907fd0e0143668b9c7ae92ddcb471dee8"} Sep 16 13:15:31.361650 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.361519 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:31.434183 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.434157 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:31.505466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.505382 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-util\") pod \"79ae829d-6052-4803-8f7a-f8baa01264ce\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " Sep 16 13:15:31.505466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.505449 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-728ff\" (UniqueName: \"kubernetes.io/projected/79ae829d-6052-4803-8f7a-f8baa01264ce-kube-api-access-728ff\") pod \"79ae829d-6052-4803-8f7a-f8baa01264ce\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " Sep 16 13:15:31.505666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.505507 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-bundle\") pod \"79ae829d-6052-4803-8f7a-f8baa01264ce\" (UID: \"79ae829d-6052-4803-8f7a-f8baa01264ce\") " Sep 16 13:15:31.506148 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.506108 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-bundle" (OuterVolumeSpecName: "bundle") pod "79ae829d-6052-4803-8f7a-f8baa01264ce" (UID: "79ae829d-6052-4803-8f7a-f8baa01264ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:31.507536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.507510 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ae829d-6052-4803-8f7a-f8baa01264ce-kube-api-access-728ff" (OuterVolumeSpecName: "kube-api-access-728ff") pod "79ae829d-6052-4803-8f7a-f8baa01264ce" (UID: "79ae829d-6052-4803-8f7a-f8baa01264ce"). InnerVolumeSpecName "kube-api-access-728ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:15:31.510990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.510971 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-util" (OuterVolumeSpecName: "util") pod "79ae829d-6052-4803-8f7a-f8baa01264ce" (UID: "79ae829d-6052-4803-8f7a-f8baa01264ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:31.606764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.606738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-util\") pod \"76a6054b-fbcc-4d42-abd8-687352722344\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " Sep 16 13:15:31.606855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.606769 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvp2l\" (UniqueName: \"kubernetes.io/projected/76a6054b-fbcc-4d42-abd8-687352722344-kube-api-access-mvp2l\") pod \"76a6054b-fbcc-4d42-abd8-687352722344\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " Sep 16 13:15:31.606855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.606820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-bundle\") pod \"76a6054b-fbcc-4d42-abd8-687352722344\" (UID: \"76a6054b-fbcc-4d42-abd8-687352722344\") " Sep 16 13:15:31.607071 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.607053 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:31.607109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.607081 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-728ff\" (UniqueName: \"kubernetes.io/projected/79ae829d-6052-4803-8f7a-f8baa01264ce-kube-api-access-728ff\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:31.607109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.607099 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79ae829d-6052-4803-8f7a-f8baa01264ce-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:31.607376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.607354 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-bundle" (OuterVolumeSpecName: "bundle") pod "76a6054b-fbcc-4d42-abd8-687352722344" (UID: "76a6054b-fbcc-4d42-abd8-687352722344"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:31.608792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.608770 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a6054b-fbcc-4d42-abd8-687352722344-kube-api-access-mvp2l" (OuterVolumeSpecName: "kube-api-access-mvp2l") pod "76a6054b-fbcc-4d42-abd8-687352722344" (UID: "76a6054b-fbcc-4d42-abd8-687352722344"). InnerVolumeSpecName "kube-api-access-mvp2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:15:31.614483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.614458 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-util" (OuterVolumeSpecName: "util") pod "76a6054b-fbcc-4d42-abd8-687352722344" (UID: "76a6054b-fbcc-4d42-abd8-687352722344"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:31.708006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.707981 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:31.708006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.708005 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvp2l\" (UniqueName: \"kubernetes.io/projected/76a6054b-fbcc-4d42-abd8-687352722344-kube-api-access-mvp2l\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:31.708149 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:31.708017 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/76a6054b-fbcc-4d42-abd8-687352722344-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:32.225090 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.225064 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" Sep 16 13:15:32.225499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.225058 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrzcc" event={"ID":"79ae829d-6052-4803-8f7a-f8baa01264ce","Type":"ContainerDied","Data":"e8887aeebf8425316bb2cfe9fc82c041055c83ba7584f1e4ffd983be23ca4b6b"} Sep 16 13:15:32.225499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.225170 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8887aeebf8425316bb2cfe9fc82c041055c83ba7584f1e4ffd983be23ca4b6b" Sep 16 13:15:32.227232 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.227204 2581 generic.go:356] "Generic (PLEG): container finished" podID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerID="c52adc34aa3c25693ae1386f9f637080bea99b3b157f3833549aaac1fd94abb5" exitCode=0 Sep 16 13:15:32.227349 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.227303 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" event={"ID":"b8df45d3-81fa-46bd-8eb4-4464949235de","Type":"ContainerDied","Data":"c52adc34aa3c25693ae1386f9f637080bea99b3b157f3833549aaac1fd94abb5"} Sep 16 13:15:32.229174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.229149 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" Sep 16 13:15:32.229174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.229163 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqhfwc" event={"ID":"76a6054b-fbcc-4d42-abd8-687352722344","Type":"ContainerDied","Data":"a46e36f6521168ed9f8a271b44284ff250e5e1d2f7ba9814d5be763a21b33df9"} Sep 16 13:15:32.229308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.229201 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a46e36f6521168ed9f8a271b44284ff250e5e1d2f7ba9814d5be763a21b33df9" Sep 16 13:15:32.360420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.360398 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:32.514665 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.514584 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-bundle\") pod \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " Sep 16 13:15:32.514665 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.514616 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-util\") pod \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " Sep 16 13:15:32.514836 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.514727 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/dd449cf6-edc7-4ffd-997c-d0c1146b883a-kube-api-access-gtqvz\") pod \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\" (UID: \"dd449cf6-edc7-4ffd-997c-d0c1146b883a\") " Sep 16 13:15:32.515141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.515110 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-bundle" (OuterVolumeSpecName: "bundle") pod "dd449cf6-edc7-4ffd-997c-d0c1146b883a" (UID: "dd449cf6-edc7-4ffd-997c-d0c1146b883a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:32.516825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.516802 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd449cf6-edc7-4ffd-997c-d0c1146b883a-kube-api-access-gtqvz" (OuterVolumeSpecName: "kube-api-access-gtqvz") pod "dd449cf6-edc7-4ffd-997c-d0c1146b883a" (UID: "dd449cf6-edc7-4ffd-997c-d0c1146b883a"). InnerVolumeSpecName "kube-api-access-gtqvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:15:32.522248 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.522223 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-util" (OuterVolumeSpecName: "util") pod "dd449cf6-edc7-4ffd-997c-d0c1146b883a" (UID: "dd449cf6-edc7-4ffd-997c-d0c1146b883a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:32.616544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.616519 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:32.616544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.616541 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd449cf6-edc7-4ffd-997c-d0c1146b883a-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:32.616544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:32.616549 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtqvz\" (UniqueName: \"kubernetes.io/projected/dd449cf6-edc7-4ffd-997c-d0c1146b883a-kube-api-access-gtqvz\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:33.235456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.235423 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" Sep 16 13:15:33.235823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.235429 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pt48c" event={"ID":"dd449cf6-edc7-4ffd-997c-d0c1146b883a","Type":"ContainerDied","Data":"ddab2654e7546962ec27aa29eb4e39e7eb8e5d8872d0c254505324aea6e929fc"} Sep 16 13:15:33.235823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.235542 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddab2654e7546962ec27aa29eb4e39e7eb8e5d8872d0c254505324aea6e929fc" Sep 16 13:15:33.361080 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.361056 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:33.423765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.423736 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-util\") pod \"b8df45d3-81fa-46bd-8eb4-4464949235de\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " Sep 16 13:15:33.423916 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.423773 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxg6z\" (UniqueName: \"kubernetes.io/projected/b8df45d3-81fa-46bd-8eb4-4464949235de-kube-api-access-cxg6z\") pod \"b8df45d3-81fa-46bd-8eb4-4464949235de\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " Sep 16 13:15:33.423916 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.423826 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-bundle\") pod \"b8df45d3-81fa-46bd-8eb4-4464949235de\" (UID: \"b8df45d3-81fa-46bd-8eb4-4464949235de\") " Sep 16 13:15:33.424376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.424350 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-bundle" (OuterVolumeSpecName: "bundle") pod "b8df45d3-81fa-46bd-8eb4-4464949235de" (UID: "b8df45d3-81fa-46bd-8eb4-4464949235de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:33.425790 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.425768 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8df45d3-81fa-46bd-8eb4-4464949235de-kube-api-access-cxg6z" (OuterVolumeSpecName: "kube-api-access-cxg6z") pod "b8df45d3-81fa-46bd-8eb4-4464949235de" (UID: "b8df45d3-81fa-46bd-8eb4-4464949235de"). InnerVolumeSpecName "kube-api-access-cxg6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:15:33.428928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.428908 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-util" (OuterVolumeSpecName: "util") pod "b8df45d3-81fa-46bd-8eb4-4464949235de" (UID: "b8df45d3-81fa-46bd-8eb4-4464949235de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:15:33.524987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.524931 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-util\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:33.524987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.524952 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxg6z\" (UniqueName: \"kubernetes.io/projected/b8df45d3-81fa-46bd-8eb4-4464949235de-kube-api-access-cxg6z\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:33.524987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:33.524963 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b8df45d3-81fa-46bd-8eb4-4464949235de-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:34.242632 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:34.242603 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" Sep 16 13:15:34.243099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:34.242632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6j9c65" event={"ID":"b8df45d3-81fa-46bd-8eb4-4464949235de","Type":"ContainerDied","Data":"380e2ed067288db3d6a71cd67c30be3b46c661ab7efea4df219ea9e4c93af754"} Sep 16 13:15:34.243099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:34.242677 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380e2ed067288db3d6a71cd67c30be3b46c661ab7efea4df219ea9e4c93af754" Sep 16 13:15:39.169607 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:39.169569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:41.558854 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:41.558813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:41.974059 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:41.974021 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:43.418420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:43.418393 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:44.552812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:44.552779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:44.827302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:44.827191 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:46.181308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181247 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8"] Sep 16 13:15:46.181772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181717 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="extract" Sep 16 13:15:46.181772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181734 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="extract" Sep 16 13:15:46.181772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181750 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="util" Sep 16 13:15:46.181772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181757 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="util" Sep 16 13:15:46.181772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181768 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="extract" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181777 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="extract" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181786 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="extract" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181794 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="extract" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181807 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181816 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181830 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="util" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181838 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="util" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181849 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181857 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181867 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="util" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181874 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="util" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181882 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181890 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181898 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181908 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="pull" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181930 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="util" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181939 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="util" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181946 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="extract" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.181955 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="extract" Sep 16 13:15:46.182031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.182031 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a6054b-fbcc-4d42-abd8-687352722344" containerName="extract" Sep 16 13:15:46.182947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.182044 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ae829d-6052-4803-8f7a-f8baa01264ce" containerName="extract" Sep 16 13:15:46.182947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.182057 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd449cf6-edc7-4ffd-997c-d0c1146b883a" containerName="extract" Sep 16 13:15:46.182947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.182066 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8df45d3-81fa-46bd-8eb4-4464949235de" containerName="extract" Sep 16 13:15:46.186628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.186608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:15:46.189437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.189414 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 13:15:46.190133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.190118 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4c6fd\"" Sep 16 13:15:46.190177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.190122 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 13:15:46.197209 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.197187 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8"] Sep 16 13:15:46.220076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.220045 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:46.232342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.232317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq846\" (UniqueName: \"kubernetes.io/projected/5f22116a-cc8f-4548-864a-9f8aebc9a9a9-kube-api-access-cq846\") pod \"limitador-operator-controller-manager-74495c69f8-8fhg8\" (UID: \"5f22116a-cc8f-4548-864a-9f8aebc9a9a9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:15:46.333462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.333426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq846\" (UniqueName: \"kubernetes.io/projected/5f22116a-cc8f-4548-864a-9f8aebc9a9a9-kube-api-access-cq846\") pod \"limitador-operator-controller-manager-74495c69f8-8fhg8\" (UID: \"5f22116a-cc8f-4548-864a-9f8aebc9a9a9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:15:46.345807 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.345775 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq846\" (UniqueName: \"kubernetes.io/projected/5f22116a-cc8f-4548-864a-9f8aebc9a9a9-kube-api-access-cq846\") pod \"limitador-operator-controller-manager-74495c69f8-8fhg8\" (UID: \"5f22116a-cc8f-4548-864a-9f8aebc9a9a9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:15:46.497200 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.497162 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:15:46.750564 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.750480 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:46.771504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:46.771483 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8"] Sep 16 13:15:46.773138 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:15:46.773115 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f22116a_cc8f_4548_864a_9f8aebc9a9a9.slice/crio-6681289ac84eeb73907f7e7430a15dfaebbc2833d1489e2c608f84c7efa37afe WatchSource:0}: Error finding container 6681289ac84eeb73907f7e7430a15dfaebbc2833d1489e2c608f84c7efa37afe: Status 404 returned error can't find the container with id 6681289ac84eeb73907f7e7430a15dfaebbc2833d1489e2c608f84c7efa37afe Sep 16 13:15:47.032923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:47.032845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:47.165730 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:47.165693 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:47.277339 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:47.277302 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:47.309462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:47.309383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" event={"ID":"5f22116a-cc8f-4548-864a-9f8aebc9a9a9","Type":"ContainerStarted","Data":"6681289ac84eeb73907f7e7430a15dfaebbc2833d1489e2c608f84c7efa37afe"} Sep 16 13:15:47.556484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:47.556453 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:47.658929 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:47.658847 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:52.338477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:52.338441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" event={"ID":"5f22116a-cc8f-4548-864a-9f8aebc9a9a9","Type":"ContainerStarted","Data":"8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1"} Sep 16 13:15:52.338844 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:52.338582 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:15:52.356746 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:52.356700 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" podStartSLOduration=1.197046577 podStartE2EDuration="6.356685927s" totalImagesPullingTime="5.15963935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:52.355187811 +0000 UTC m=+437.474914359" watchObservedRunningTime="2026-09-16 13:15:52.356685927 +0000 UTC m=+437.476412476" Sep 16 13:15:52.762753 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:52.762704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:52.854723 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:52.854682 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:54.342841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.342801 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf5d44cb9-tphb5" podUID="0cb5e70c-bee6-4e63-8d74-718b33263fa4" containerName="console" containerID="cri-o://49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47" gracePeriod=15 Sep 16 13:15:54.585963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.585939 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:15:54.704460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704436 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-oauth-serving-cert\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.704635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704487 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-trusted-ca-bundle\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.704635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704506 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-oauth-config\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.704635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704532 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-config\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.704802 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704692 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-service-ca\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.704802 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704727 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-serving-cert\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.704802 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704758 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxrwj\" (UniqueName: \"kubernetes.io/projected/0cb5e70c-bee6-4e63-8d74-718b33263fa4-kube-api-access-fxrwj\") pod \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\" (UID: \"0cb5e70c-bee6-4e63-8d74-718b33263fa4\") " Sep 16 13:15:54.705015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.704991 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:15:54.705093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.705061 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:15:54.705138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.705114 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-service-ca" (OuterVolumeSpecName: "service-ca") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:15:54.705196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.705134 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-config" (OuterVolumeSpecName: "console-config") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:15:54.706798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.706771 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:15:54.706888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.706806 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb5e70c-bee6-4e63-8d74-718b33263fa4-kube-api-access-fxrwj" (OuterVolumeSpecName: "kube-api-access-fxrwj") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "kube-api-access-fxrwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:15:54.706938 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.706918 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0cb5e70c-bee6-4e63-8d74-718b33263fa4" (UID: "0cb5e70c-bee6-4e63-8d74-718b33263fa4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:15:54.805899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805861 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-oauth-serving-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:54.805899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805891 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-trusted-ca-bundle\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:54.805899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805903 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-oauth-config\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:54.806139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805916 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-config\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:54.806139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805928 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb5e70c-bee6-4e63-8d74-718b33263fa4-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:54.806139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805938 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb5e70c-bee6-4e63-8d74-718b33263fa4-console-serving-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:54.806139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:54.805949 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxrwj\" (UniqueName: \"kubernetes.io/projected/0cb5e70c-bee6-4e63-8d74-718b33263fa4-kube-api-access-fxrwj\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:15:55.355996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.355959 2581 generic.go:356] "Generic (PLEG): container finished" podID="0cb5e70c-bee6-4e63-8d74-718b33263fa4" containerID="49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47" exitCode=0 Sep 16 13:15:55.356472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.356024 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf5d44cb9-tphb5" Sep 16 13:15:55.356472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.356048 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf5d44cb9-tphb5" event={"ID":"0cb5e70c-bee6-4e63-8d74-718b33263fa4","Type":"ContainerDied","Data":"49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47"} Sep 16 13:15:55.356472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.356138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf5d44cb9-tphb5" event={"ID":"0cb5e70c-bee6-4e63-8d74-718b33263fa4","Type":"ContainerDied","Data":"0db7220f11e941310910def4722514d7cf3394162067449001dff1baf810914f"} Sep 16 13:15:55.356472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.356169 2581 scope.go:117] "RemoveContainer" containerID="49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47" Sep 16 13:15:55.365356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.365337 2581 scope.go:117] "RemoveContainer" containerID="49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47" Sep 16 13:15:55.365620 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:15:55.365601 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47\": container with ID starting with 49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47 not found: ID does not exist" containerID="49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47" Sep 16 13:15:55.365694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.365630 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47"} err="failed to get container status \"49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47\": rpc error: code = NotFound desc = could not find container \"49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47\": container with ID starting with 49fad8e6e819522f75b0b413b4f656aa18a1c0743a9b5122ba0baa0167a04b47 not found: ID does not exist" Sep 16 13:15:55.377820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.377795 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf5d44cb9-tphb5"] Sep 16 13:15:55.386080 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:55.386058 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf5d44cb9-tphb5"] Sep 16 13:15:57.386097 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:57.386062 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb5e70c-bee6-4e63-8d74-718b33263fa4" path="/var/lib/kubelet/pods/0cb5e70c-bee6-4e63-8d74-718b33263fa4/volumes" Sep 16 13:15:57.868469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:57.868368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:58.141566 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:58.141483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:58.364829 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:58.364802 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:58.450067 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:58.450036 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:58.781862 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:58.781782 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:58.880692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:58.880662 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:59.401773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:59.401739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:15:59.461035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:15:59.460999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:00.383005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:00.382977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:00.485737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:00.485709 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:03.345628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:03.345596 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:05.140077 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:05.140050 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:05.244366 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:05.244337 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:05.376073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:05.376035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:08.917930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.917889 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49"] Sep 16 13:16:08.918420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.918359 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb5e70c-bee6-4e63-8d74-718b33263fa4" containerName="console" Sep 16 13:16:08.918420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.918377 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb5e70c-bee6-4e63-8d74-718b33263fa4" containerName="console" Sep 16 13:16:08.918538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.918469 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb5e70c-bee6-4e63-8d74-718b33263fa4" containerName="console" Sep 16 13:16:08.922818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.922798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:08.925403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.925380 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jmzkj\"" Sep 16 13:16:08.934601 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:08.934578 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49"] Sep 16 13:16:09.024914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.024888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgph\" (UniqueName: \"kubernetes.io/projected/2a073eba-2277-4c54-9f3b-b6df4541e3e8-kube-api-access-pzgph\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.025056 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.024973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a073eba-2277-4c54-9f3b-b6df4541e3e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.028657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.028633 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49"] Sep 16 13:16:09.028867 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.028849 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-pzgph], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" podUID="2a073eba-2277-4c54-9f3b-b6df4541e3e8" Sep 16 13:16:09.067441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.067413 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq"] Sep 16 13:16:09.070703 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.070688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.084019 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.083996 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq"] Sep 16 13:16:09.125367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.125338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgph\" (UniqueName: \"kubernetes.io/projected/2a073eba-2277-4c54-9f3b-b6df4541e3e8-kube-api-access-pzgph\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.125717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.125387 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5r4h\" (UniqueName: \"kubernetes.io/projected/e3e54d68-8696-4fce-928a-ea7282ae41e1-kube-api-access-c5r4h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.125717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.125432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3e54d68-8696-4fce-928a-ea7282ae41e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.125717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.125456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a073eba-2277-4c54-9f3b-b6df4541e3e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.125918 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.125778 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a073eba-2277-4c54-9f3b-b6df4541e3e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.134667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.134643 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgph\" (UniqueName: \"kubernetes.io/projected/2a073eba-2277-4c54-9f3b-b6df4541e3e8-kube-api-access-pzgph\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.226302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.226256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5r4h\" (UniqueName: \"kubernetes.io/projected/e3e54d68-8696-4fce-928a-ea7282ae41e1-kube-api-access-c5r4h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.226449 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.226321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3e54d68-8696-4fce-928a-ea7282ae41e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.226641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.226625 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3e54d68-8696-4fce-928a-ea7282ae41e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.235618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.235594 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5r4h\" (UniqueName: \"kubernetes.io/projected/e3e54d68-8696-4fce-928a-ea7282ae41e1-kube-api-access-c5r4h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.380875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.380836 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.426968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.426936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.434532 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.434503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:09.528217 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.528136 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a073eba-2277-4c54-9f3b-b6df4541e3e8-extensions-socket-volume\") pod \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " Sep 16 13:16:09.528217 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.528171 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzgph\" (UniqueName: \"kubernetes.io/projected/2a073eba-2277-4c54-9f3b-b6df4541e3e8-kube-api-access-pzgph\") pod \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\" (UID: \"2a073eba-2277-4c54-9f3b-b6df4541e3e8\") " Sep 16 13:16:09.528444 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.528422 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a073eba-2277-4c54-9f3b-b6df4541e3e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a073eba-2277-4c54-9f3b-b6df4541e3e8" (UID: "2a073eba-2277-4c54-9f3b-b6df4541e3e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:16:09.530275 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.530260 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a073eba-2277-4c54-9f3b-b6df4541e3e8-kube-api-access-pzgph" (OuterVolumeSpecName: "kube-api-access-pzgph") pod "2a073eba-2277-4c54-9f3b-b6df4541e3e8" (UID: "2a073eba-2277-4c54-9f3b-b6df4541e3e8"). InnerVolumeSpecName "kube-api-access-pzgph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:16:09.606556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.606525 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49"] Sep 16 13:16:09.610020 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.609993 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49"] Sep 16 13:16:09.629117 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.629096 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a073eba-2277-4c54-9f3b-b6df4541e3e8-extensions-socket-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:09.629117 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.629117 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzgph\" (UniqueName: \"kubernetes.io/projected/2a073eba-2277-4c54-9f3b-b6df4541e3e8-kube-api-access-pzgph\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:09.643124 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.643095 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq"] Sep 16 13:16:09.651438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.651412 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq"] Sep 16 13:16:09.667300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.667264 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8"] Sep 16 13:16:09.667506 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.667475 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" containerName="manager" containerID="cri-o://8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1" gracePeriod=2 Sep 16 13:16:09.672718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.672690 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc"] Sep 16 13:16:09.682406 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.681390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:09.688103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.687268 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8"] Sep 16 13:16:09.688205 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.688167 2581 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5f22116a-cc8f-4548-864a-9f8aebc9a9a9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-8fhg8\": pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:09.693454 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.693420 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc"] Sep 16 13:16:09.698535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.698512 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj"] Sep 16 13:16:09.698820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.698807 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" containerName="manager" Sep 16 13:16:09.698892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.698821 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" containerName="manager" Sep 16 13:16:09.698892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.698888 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" containerName="manager" Sep 16 13:16:09.701938 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.701913 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:09.712293 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.710406 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:09.713039 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.713008 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:09.715555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.715529 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj"] Sep 16 13:16:09.788666 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.788634 2581 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 16 13:16:09.788666 ip-10-0-133-73 kubenswrapper[2581]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-w5dcq_kuadrant-system_e3e54d68-8696-4fce-928a-ea7282ae41e1_0(0a7aa56bef7b0524aea1878c9b3d85adbb8797610b6c4e21220ed3f1155ac42a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-w5dcq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0a7aa56bef7b0524aea1878c9b3d85adbb8797610b6c4e21220ed3f1155ac42a" Netns:"/var/run/netns/ffbc27b1-9f19-4dd1-81a1-30120732a556" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-w5dcq;K8S_POD_INFRA_CONTAINER_ID=0a7aa56bef7b0524aea1878c9b3d85adbb8797610b6c4e21220ed3f1155ac42a;K8S_POD_UID=e3e54d68-8696-4fce-928a-ea7282ae41e1" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq/e3e54d68-8696-4fce-928a-ea7282ae41e1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-7978cbdfbc-w5dcq in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-7978cbdfbc-w5dcq in out of cluster comm: status update failed for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq: pods "kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" not found Sep 16 13:16:09.788666 ip-10-0-133-73 kubenswrapper[2581]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 16 13:16:09.788666 ip-10-0-133-73 kubenswrapper[2581]: > Sep 16 13:16:09.788890 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.788696 2581 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 16 13:16:09.788890 ip-10-0-133-73 kubenswrapper[2581]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-w5dcq_kuadrant-system_e3e54d68-8696-4fce-928a-ea7282ae41e1_0(0a7aa56bef7b0524aea1878c9b3d85adbb8797610b6c4e21220ed3f1155ac42a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-w5dcq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0a7aa56bef7b0524aea1878c9b3d85adbb8797610b6c4e21220ed3f1155ac42a" Netns:"/var/run/netns/ffbc27b1-9f19-4dd1-81a1-30120732a556" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-w5dcq;K8S_POD_INFRA_CONTAINER_ID=0a7aa56bef7b0524aea1878c9b3d85adbb8797610b6c4e21220ed3f1155ac42a;K8S_POD_UID=e3e54d68-8696-4fce-928a-ea7282ae41e1" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq/e3e54d68-8696-4fce-928a-ea7282ae41e1]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-7978cbdfbc-w5dcq in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-7978cbdfbc-w5dcq in out of cluster comm: status update failed for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq: pods "kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" not found Sep 16 13:16:09.788890 ip-10-0-133-73 kubenswrapper[2581]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 16 13:16:09.788890 ip-10-0-133-73 kubenswrapper[2581]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:09.830894 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.830861 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtrl\" (UniqueName: \"kubernetes.io/projected/24efcd0c-c733-4f6b-bae7-e18ac2953f20-kube-api-access-cjtrl\") pod \"limitador-operator-controller-manager-74495c69f8-744dj\" (UID: \"24efcd0c-c733-4f6b-bae7-e18ac2953f20\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:09.831068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.830912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dd7100-457a-4a16-af24-08d109246113-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6kmgc\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:09.831068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.831028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58h6q\" (UniqueName: \"kubernetes.io/projected/a5dd7100-457a-4a16-af24-08d109246113-kube-api-access-58h6q\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6kmgc\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:09.893988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.893963 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:09.896205 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:09.896178 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:09.931622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.931589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtrl\" (UniqueName: \"kubernetes.io/projected/24efcd0c-c733-4f6b-bae7-e18ac2953f20-kube-api-access-cjtrl\") pod \"limitador-operator-controller-manager-74495c69f8-744dj\" (UID: \"24efcd0c-c733-4f6b-bae7-e18ac2953f20\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:09.931985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.931634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dd7100-457a-4a16-af24-08d109246113-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6kmgc\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:09.931985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.931666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58h6q\" (UniqueName: \"kubernetes.io/projected/a5dd7100-457a-4a16-af24-08d109246113-kube-api-access-58h6q\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6kmgc\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:09.931985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.931977 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dd7100-457a-4a16-af24-08d109246113-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6kmgc\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:09.940039 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.940012 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtrl\" (UniqueName: \"kubernetes.io/projected/24efcd0c-c733-4f6b-bae7-e18ac2953f20-kube-api-access-cjtrl\") pod \"limitador-operator-controller-manager-74495c69f8-744dj\" (UID: \"24efcd0c-c733-4f6b-bae7-e18ac2953f20\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:09.940039 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:09.940032 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58h6q\" (UniqueName: \"kubernetes.io/projected/a5dd7100-457a-4a16-af24-08d109246113-kube-api-access-58h6q\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-6kmgc\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:10.032336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.032304 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq846\" (UniqueName: \"kubernetes.io/projected/5f22116a-cc8f-4548-864a-9f8aebc9a9a9-kube-api-access-cq846\") pod \"5f22116a-cc8f-4548-864a-9f8aebc9a9a9\" (UID: \"5f22116a-cc8f-4548-864a-9f8aebc9a9a9\") " Sep 16 13:16:10.034375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.034352 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f22116a-cc8f-4548-864a-9f8aebc9a9a9-kube-api-access-cq846" (OuterVolumeSpecName: "kube-api-access-cq846") pod "5f22116a-cc8f-4548-864a-9f8aebc9a9a9" (UID: "5f22116a-cc8f-4548-864a-9f8aebc9a9a9"). InnerVolumeSpecName "kube-api-access-cq846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:16:10.044008 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.043960 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:10.050918 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.050902 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:10.132959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.132929 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq846\" (UniqueName: \"kubernetes.io/projected/5f22116a-cc8f-4548-864a-9f8aebc9a9a9-kube-api-access-cq846\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:10.339541 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.339499 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj"] Sep 16 13:16:10.341477 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:16:10.341445 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dd7100_457a_4a16_af24_08d109246113.slice/crio-632d984e65acc06e6a54dbb2defa99cab800668f6bb1004f2af55fcb0f2481d3 WatchSource:0}: Error finding container 632d984e65acc06e6a54dbb2defa99cab800668f6bb1004f2af55fcb0f2481d3: Status 404 returned error can't find the container with id 632d984e65acc06e6a54dbb2defa99cab800668f6bb1004f2af55fcb0f2481d3 Sep 16 13:16:10.342190 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:16:10.342163 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24efcd0c_c733_4f6b_bae7_e18ac2953f20.slice/crio-8bed599310bcc245ac423e46a3af54727715405548759b83bb8469afc1ec1418 WatchSource:0}: Error finding container 8bed599310bcc245ac423e46a3af54727715405548759b83bb8469afc1ec1418: Status 404 returned error can't find the container with id 8bed599310bcc245ac423e46a3af54727715405548759b83bb8469afc1ec1418 Sep 16 13:16:10.342190 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.342170 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc"] Sep 16 13:16:10.432890 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.432859 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" event={"ID":"a5dd7100-457a-4a16-af24-08d109246113","Type":"ContainerStarted","Data":"632d984e65acc06e6a54dbb2defa99cab800668f6bb1004f2af55fcb0f2481d3"} Sep 16 13:16:10.434220 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.434194 2581 generic.go:356] "Generic (PLEG): container finished" podID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" containerID="8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1" exitCode=0 Sep 16 13:16:10.434364 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.434231 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:10.434364 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.434305 2581 scope.go:117] "RemoveContainer" containerID="8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1" Sep 16 13:16:10.436111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.436090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:10.436212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.436120 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" event={"ID":"24efcd0c-c733-4f6b-bae7-e18ac2953f20","Type":"ContainerStarted","Data":"1a82d380cea778415fdb5c4b8ec94b067e9b56af6995888d96df49627c626539"} Sep 16 13:16:10.436212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.436169 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" event={"ID":"24efcd0c-c733-4f6b-bae7-e18ac2953f20","Type":"ContainerStarted","Data":"8bed599310bcc245ac423e46a3af54727715405548759b83bb8469afc1ec1418"} Sep 16 13:16:10.436212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.436198 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:10.436399 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.436230 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:10.437422 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.437401 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:10.439391 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.439364 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="2a073eba-2277-4c54-9f3b-b6df4541e3e8" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:10.441380 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.441356 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:10.443196 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.443174 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="e3e54d68-8696-4fce-928a-ea7282ae41e1" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:10.444246 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.444227 2581 scope.go:117] "RemoveContainer" containerID="8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1" Sep 16 13:16:10.444522 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.444501 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1\": container with ID starting with 8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1 not found: ID does not exist" containerID="8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1" Sep 16 13:16:10.444592 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.444533 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1"} err="failed to get container status \"8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1\": rpc error: code = NotFound desc = could not find container \"8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1\": container with ID starting with 8bac005c55e6c8394bd3627b5287dd74dcfaa932ca0bcd1ddb16758953dff9b1 not found: ID does not exist" Sep 16 13:16:10.444592 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.444568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:10.485309 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.485247 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" podStartSLOduration=1.485234334 podStartE2EDuration="1.485234334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:10.483360599 +0000 UTC m=+455.603087172" watchObservedRunningTime="2026-09-16 13:16:10.485234334 +0000 UTC m=+455.604960883" Sep 16 13:16:10.491493 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.491468 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="e3e54d68-8696-4fce-928a-ea7282ae41e1" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:10.493658 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.493634 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzz49\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="2a073eba-2277-4c54-9f3b-b6df4541e3e8" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzz49" Sep 16 13:16:10.496608 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:10.496584 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8fhg8\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8fhg8" Sep 16 13:16:10.536459 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.536433 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5r4h\" (UniqueName: \"kubernetes.io/projected/e3e54d68-8696-4fce-928a-ea7282ae41e1-kube-api-access-c5r4h\") pod \"e3e54d68-8696-4fce-928a-ea7282ae41e1\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " Sep 16 13:16:10.536547 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.536528 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3e54d68-8696-4fce-928a-ea7282ae41e1-extensions-socket-volume\") pod \"e3e54d68-8696-4fce-928a-ea7282ae41e1\" (UID: \"e3e54d68-8696-4fce-928a-ea7282ae41e1\") " Sep 16 13:16:10.536763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.536742 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e54d68-8696-4fce-928a-ea7282ae41e1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e3e54d68-8696-4fce-928a-ea7282ae41e1" (UID: "e3e54d68-8696-4fce-928a-ea7282ae41e1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:16:10.538458 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.538432 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e54d68-8696-4fce-928a-ea7282ae41e1-kube-api-access-c5r4h" (OuterVolumeSpecName: "kube-api-access-c5r4h") pod "e3e54d68-8696-4fce-928a-ea7282ae41e1" (UID: "e3e54d68-8696-4fce-928a-ea7282ae41e1"). InnerVolumeSpecName "kube-api-access-c5r4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:16:10.637602 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.637529 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3e54d68-8696-4fce-928a-ea7282ae41e1-extensions-socket-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:10.637602 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:10.637551 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5r4h\" (UniqueName: \"kubernetes.io/projected/e3e54d68-8696-4fce-928a-ea7282ae41e1-kube-api-access-c5r4h\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:11.386298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:11.386247 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a073eba-2277-4c54-9f3b-b6df4541e3e8" path="/var/lib/kubelet/pods/2a073eba-2277-4c54-9f3b-b6df4541e3e8/volumes" Sep 16 13:16:11.386667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:11.386503 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f22116a-cc8f-4548-864a-9f8aebc9a9a9" path="/var/lib/kubelet/pods/5f22116a-cc8f-4548-864a-9f8aebc9a9a9/volumes" Sep 16 13:16:11.386817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:11.386804 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e54d68-8696-4fce-928a-ea7282ae41e1" path="/var/lib/kubelet/pods/e3e54d68-8696-4fce-928a-ea7282ae41e1/volumes" Sep 16 13:16:11.442073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:11.442044 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:11.445841 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:11.445812 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-w5dcq\" is forbidden: User \"system:node:ip-10-0-133-73.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-73.ec2.internal' and this object" podUID="e3e54d68-8696-4fce-928a-ea7282ae41e1" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-w5dcq" Sep 16 13:16:12.589096 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:12.589055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:12.686602 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:12.686570 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:15.467517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:15.467478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" event={"ID":"a5dd7100-457a-4a16-af24-08d109246113","Type":"ContainerStarted","Data":"03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5"} Sep 16 13:16:15.467898 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:15.467547 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:15.500499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:15.500447 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" podStartSLOduration=2.058568242 podStartE2EDuration="6.500431347s" totalImagesPullingTime="4.441863105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:15.499920067 +0000 UTC m=+460.619646677" watchObservedRunningTime="2026-09-16 13:16:15.500431347 +0000 UTC m=+460.620157904" Sep 16 13:16:17.236803 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:17.236773 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:17.371011 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:17.370960 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:17.616776 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:17.616675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:17.715949 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:17.715913 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:21.010389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:21.010357 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:21.111837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:21.111810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:21.444868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:21.444789 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-744dj" Sep 16 13:16:24.917035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:24.917003 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:25.662551 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:25.662509 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:25.762376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:25.762342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:26.474412 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.474383 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:26.736383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.736306 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7"] Sep 16 13:16:26.744991 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.744972 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:26.753683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.753660 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7"] Sep 16 13:16:26.877538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.877503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93b15a2-5658-4cf3-b16c-6a4abfe07b44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2l9p7\" (UID: \"f93b15a2-5658-4cf3-b16c-6a4abfe07b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:26.877713 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.877566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc846\" (UniqueName: \"kubernetes.io/projected/f93b15a2-5658-4cf3-b16c-6a4abfe07b44-kube-api-access-jc846\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2l9p7\" (UID: \"f93b15a2-5658-4cf3-b16c-6a4abfe07b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:26.978786 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.978755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93b15a2-5658-4cf3-b16c-6a4abfe07b44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2l9p7\" (UID: \"f93b15a2-5658-4cf3-b16c-6a4abfe07b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:26.978970 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.978816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc846\" (UniqueName: \"kubernetes.io/projected/f93b15a2-5658-4cf3-b16c-6a4abfe07b44-kube-api-access-jc846\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2l9p7\" (UID: \"f93b15a2-5658-4cf3-b16c-6a4abfe07b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:26.979170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.979150 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93b15a2-5658-4cf3-b16c-6a4abfe07b44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2l9p7\" (UID: \"f93b15a2-5658-4cf3-b16c-6a4abfe07b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:26.991162 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:26.991104 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc846\" (UniqueName: \"kubernetes.io/projected/f93b15a2-5658-4cf3-b16c-6a4abfe07b44-kube-api-access-jc846\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2l9p7\" (UID: \"f93b15a2-5658-4cf3-b16c-6a4abfe07b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:27.055986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:27.055951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:27.350257 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:27.350233 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7"] Sep 16 13:16:27.351720 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:16:27.351694 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93b15a2_5658_4cf3_b16c_6a4abfe07b44.slice/crio-ef6b45a8308c89d634d7e7974f3caf9ed29c737909fbbaf7ec6e220784b24557 WatchSource:0}: Error finding container ef6b45a8308c89d634d7e7974f3caf9ed29c737909fbbaf7ec6e220784b24557: Status 404 returned error can't find the container with id ef6b45a8308c89d634d7e7974f3caf9ed29c737909fbbaf7ec6e220784b24557 Sep 16 13:16:27.530155 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:27.530113 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" event={"ID":"f93b15a2-5658-4cf3-b16c-6a4abfe07b44","Type":"ContainerStarted","Data":"ee628c7c0efce4dca659146ef2bc33db61acf7b34e6f87edc332033fd6d7c46f"} Sep 16 13:16:27.530636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:27.530263 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:27.530636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:27.530324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" event={"ID":"f93b15a2-5658-4cf3-b16c-6a4abfe07b44","Type":"ContainerStarted","Data":"ef6b45a8308c89d634d7e7974f3caf9ed29c737909fbbaf7ec6e220784b24557"} Sep 16 13:16:27.547021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:27.546979 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" podStartSLOduration=1.546965008 podStartE2EDuration="1.546965008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:27.545443229 +0000 UTC m=+472.665169778" watchObservedRunningTime="2026-09-16 13:16:27.546965008 +0000 UTC m=+472.666691556" Sep 16 13:16:38.537731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.537701 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2l9p7" Sep 16 13:16:38.590823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.590795 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc"] Sep 16 13:16:38.591003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.590968 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" podUID="a5dd7100-457a-4a16-af24-08d109246113" containerName="manager" containerID="cri-o://03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5" gracePeriod=10 Sep 16 13:16:38.834069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.834041 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:38.868873 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.868846 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58h6q\" (UniqueName: \"kubernetes.io/projected/a5dd7100-457a-4a16-af24-08d109246113-kube-api-access-58h6q\") pod \"a5dd7100-457a-4a16-af24-08d109246113\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " Sep 16 13:16:38.869010 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.868895 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dd7100-457a-4a16-af24-08d109246113-extensions-socket-volume\") pod \"a5dd7100-457a-4a16-af24-08d109246113\" (UID: \"a5dd7100-457a-4a16-af24-08d109246113\") " Sep 16 13:16:38.869264 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.869242 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dd7100-457a-4a16-af24-08d109246113-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a5dd7100-457a-4a16-af24-08d109246113" (UID: "a5dd7100-457a-4a16-af24-08d109246113"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:16:38.870882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.870859 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dd7100-457a-4a16-af24-08d109246113-kube-api-access-58h6q" (OuterVolumeSpecName: "kube-api-access-58h6q") pod "a5dd7100-457a-4a16-af24-08d109246113" (UID: "a5dd7100-457a-4a16-af24-08d109246113"). InnerVolumeSpecName "kube-api-access-58h6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:16:38.970181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.970155 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58h6q\" (UniqueName: \"kubernetes.io/projected/a5dd7100-457a-4a16-af24-08d109246113-kube-api-access-58h6q\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:38.970181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:38.970175 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dd7100-457a-4a16-af24-08d109246113-extensions-socket-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:16:39.592605 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.592569 2581 generic.go:356] "Generic (PLEG): container finished" podID="a5dd7100-457a-4a16-af24-08d109246113" containerID="03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5" exitCode=0 Sep 16 13:16:39.593034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.592631 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" Sep 16 13:16:39.593034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.592655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" event={"ID":"a5dd7100-457a-4a16-af24-08d109246113","Type":"ContainerDied","Data":"03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5"} Sep 16 13:16:39.593034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.592711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc" event={"ID":"a5dd7100-457a-4a16-af24-08d109246113","Type":"ContainerDied","Data":"632d984e65acc06e6a54dbb2defa99cab800668f6bb1004f2af55fcb0f2481d3"} Sep 16 13:16:39.593034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.592734 2581 scope.go:117] "RemoveContainer" containerID="03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5" Sep 16 13:16:39.601070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.601052 2581 scope.go:117] "RemoveContainer" containerID="03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5" Sep 16 13:16:39.601350 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:16:39.601329 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5\": container with ID starting with 03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5 not found: ID does not exist" containerID="03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5" Sep 16 13:16:39.601451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.601355 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5"} err="failed to get container status \"03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5\": rpc error: code = NotFound desc = could not find container \"03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5\": container with ID starting with 03eb4b916a3062aeb9fcee33b01d21e668e2c9b18d8503c1994d82b6a036a1b5 not found: ID does not exist" Sep 16 13:16:39.619646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.619621 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc"] Sep 16 13:16:39.624816 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:39.624796 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-6kmgc"] Sep 16 13:16:41.386411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:41.386382 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5dd7100-457a-4a16-af24-08d109246113" path="/var/lib/kubelet/pods/a5dd7100-457a-4a16-af24-08d109246113/volumes" Sep 16 13:16:43.021158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:43.021124 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:43.455818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:43.455786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:47.651417 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:47.651371 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:47.759868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:47.759832 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:47.860791 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:47.860758 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:48.510568 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:48.510536 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:16:48.610320 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:16:48.610262 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:07.087687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:07.087647 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:07.190251 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:07.190215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:07.601417 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:07.601385 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:15.803058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:15.803030 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:19.964449 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:19.964409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:20.179520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:20.179479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:28.238220 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.238138 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:28.290633 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.290602 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-689998fb9f-slflh"] Sep 16 13:17:28.291007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.290990 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5dd7100-457a-4a16-af24-08d109246113" containerName="manager" Sep 16 13:17:28.291109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.291009 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5dd7100-457a-4a16-af24-08d109246113" containerName="manager" Sep 16 13:17:28.291169 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.291131 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5dd7100-457a-4a16-af24-08d109246113" containerName="manager" Sep 16 13:17:28.294295 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.294261 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.302418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.302397 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 13:17:28.303455 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.303436 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 13:17:28.303551 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.303464 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hqshj\"" Sep 16 13:17:28.304355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.304336 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 13:17:28.322296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.322264 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-689998fb9f-slflh"] Sep 16 13:17:28.489659 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.489580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-webhook-cert\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.489659 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.489610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-metrics-certs\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.489835 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.489687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmprd\" (UniqueName: \"kubernetes.io/projected/e2395013-bcef-4fa6-9176-f339b2967ce6-kube-api-access-kmprd\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.590337 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.590302 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-webhook-cert\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.590337 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.590337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-metrics-certs\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.590540 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.590393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmprd\" (UniqueName: \"kubernetes.io/projected/e2395013-bcef-4fa6-9176-f339b2967ce6-kube-api-access-kmprd\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.592870 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.592842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-webhook-cert\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.592990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.592975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-metrics-certs\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.599709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.599685 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmprd\" (UniqueName: \"kubernetes.io/projected/e2395013-bcef-4fa6-9176-f339b2967ce6-kube-api-access-kmprd\") pod \"maas-controller-689998fb9f-slflh\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.605462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.605428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:28.897147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.897114 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-689998fb9f-slflh"] Sep 16 13:17:28.897778 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:17:28.897755 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2395013_bcef_4fa6_9176_f339b2967ce6.slice/crio-cf87a3030e60906a67746f5b711916bb57f267c49d2bf333fb166d522a38132a WatchSource:0}: Error finding container cf87a3030e60906a67746f5b711916bb57f267c49d2bf333fb166d522a38132a: Status 404 returned error can't find the container with id cf87a3030e60906a67746f5b711916bb57f267c49d2bf333fb166d522a38132a Sep 16 13:17:28.947073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:28.947034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:29.044560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:29.044531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:29.857562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:29.857497 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689998fb9f-slflh" event={"ID":"e2395013-bcef-4fa6-9176-f339b2967ce6","Type":"ContainerStarted","Data":"cf87a3030e60906a67746f5b711916bb57f267c49d2bf333fb166d522a38132a"} Sep 16 13:17:31.631547 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:31.631515 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:31.735584 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:31.735540 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:17:31.869495 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:31.869459 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689998fb9f-slflh" event={"ID":"e2395013-bcef-4fa6-9176-f339b2967ce6","Type":"ContainerStarted","Data":"d7394ea6c8a4bc92d5af7604dfae87f3170a71a95f19dd0a05002fb10db83724"} Sep 16 13:17:31.869689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:31.869519 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:31.897781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:31.897734 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-689998fb9f-slflh" podStartSLOduration=1.356729428 podStartE2EDuration="3.89771869s" totalImagesPullingTime="2.540989262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:31.895678324 +0000 UTC m=+537.015404873" watchObservedRunningTime="2026-09-16 13:17:31.89771869 +0000 UTC m=+537.017445239" Sep 16 13:17:42.696251 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.696214 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-689998fb9f-slflh"] Sep 16 13:17:42.696692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.696428 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-689998fb9f-slflh" podUID="e2395013-bcef-4fa6-9176-f339b2967ce6" containerName="manager" containerID="cri-o://d7394ea6c8a4bc92d5af7604dfae87f3170a71a95f19dd0a05002fb10db83724" gracePeriod=30 Sep 16 13:17:42.698118 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.698097 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:42.929980 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.929949 2581 generic.go:356] "Generic (PLEG): container finished" podID="e2395013-bcef-4fa6-9176-f339b2967ce6" containerID="d7394ea6c8a4bc92d5af7604dfae87f3170a71a95f19dd0a05002fb10db83724" exitCode=0 Sep 16 13:17:42.930052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.930028 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689998fb9f-slflh" event={"ID":"e2395013-bcef-4fa6-9176-f339b2967ce6","Type":"ContainerDied","Data":"d7394ea6c8a4bc92d5af7604dfae87f3170a71a95f19dd0a05002fb10db83724"} Sep 16 13:17:42.943851 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.943832 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:42.998791 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.998772 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-metrics-certs\") pod \"e2395013-bcef-4fa6-9176-f339b2967ce6\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " Sep 16 13:17:42.998912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.998806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmprd\" (UniqueName: \"kubernetes.io/projected/e2395013-bcef-4fa6-9176-f339b2967ce6-kube-api-access-kmprd\") pod \"e2395013-bcef-4fa6-9176-f339b2967ce6\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " Sep 16 13:17:42.998912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:42.998827 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-webhook-cert\") pod \"e2395013-bcef-4fa6-9176-f339b2967ce6\" (UID: \"e2395013-bcef-4fa6-9176-f339b2967ce6\") " Sep 16 13:17:43.000906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.000879 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2395013-bcef-4fa6-9176-f339b2967ce6" (UID: "e2395013-bcef-4fa6-9176-f339b2967ce6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:17:43.001004 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.000954 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2395013-bcef-4fa6-9176-f339b2967ce6-kube-api-access-kmprd" (OuterVolumeSpecName: "kube-api-access-kmprd") pod "e2395013-bcef-4fa6-9176-f339b2967ce6" (UID: "e2395013-bcef-4fa6-9176-f339b2967ce6"). InnerVolumeSpecName "kube-api-access-kmprd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:17:43.001065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.001010 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e2395013-bcef-4fa6-9176-f339b2967ce6" (UID: "e2395013-bcef-4fa6-9176-f339b2967ce6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:17:43.099616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.099588 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-metrics-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:17:43.099616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.099613 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmprd\" (UniqueName: \"kubernetes.io/projected/e2395013-bcef-4fa6-9176-f339b2967ce6-kube-api-access-kmprd\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:17:43.099769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.099625 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2395013-bcef-4fa6-9176-f339b2967ce6-webhook-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:17:43.935948 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.935916 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689998fb9f-slflh" event={"ID":"e2395013-bcef-4fa6-9176-f339b2967ce6","Type":"ContainerDied","Data":"cf87a3030e60906a67746f5b711916bb57f267c49d2bf333fb166d522a38132a"} Sep 16 13:17:43.936342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.935977 2581 scope.go:117] "RemoveContainer" containerID="d7394ea6c8a4bc92d5af7604dfae87f3170a71a95f19dd0a05002fb10db83724" Sep 16 13:17:43.936342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.935928 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689998fb9f-slflh" Sep 16 13:17:43.957707 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.957678 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-689998fb9f-slflh"] Sep 16 13:17:43.959576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:43.959556 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-689998fb9f-slflh"] Sep 16 13:17:45.386759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:17:45.386726 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2395013-bcef-4fa6-9176-f339b2967ce6" path="/var/lib/kubelet/pods/e2395013-bcef-4fa6-9176-f339b2967ce6/volumes" Sep 16 13:18:01.929629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:01.929591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:02.046303 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:02.046231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:04.345321 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:04.345240 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:04.617437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:04.617344 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:05.903999 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:05.903955 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.048408 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.048363 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.187524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.187438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.423365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.423326 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.606601 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.606560 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.721751 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.721716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.786700 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.786669 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn"] Sep 16 13:18:06.787018 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.787006 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2395013-bcef-4fa6-9176-f339b2967ce6" containerName="manager" Sep 16 13:18:06.787063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.787020 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2395013-bcef-4fa6-9176-f339b2967ce6" containerName="manager" Sep 16 13:18:06.787098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.787075 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2395013-bcef-4fa6-9176-f339b2967ce6" containerName="manager" Sep 16 13:18:06.792658 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.792638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:06.795660 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.795636 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-glgpg\"" Sep 16 13:18:06.795911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.795896 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 16 13:18:06.807773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.807748 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn"] Sep 16 13:18:06.869617 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.869505 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:06.897568 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.897535 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4d6f37-8a71-4355-937a-be27626ab4d5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-l46wn\" (UID: \"8e4d6f37-8a71-4355-937a-be27626ab4d5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:06.897747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.897598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n477z\" (UniqueName: \"kubernetes.io/projected/8e4d6f37-8a71-4355-937a-be27626ab4d5-kube-api-access-n477z\") pod \"payload-pre-processing-59fbc47bf8-l46wn\" (UID: \"8e4d6f37-8a71-4355-937a-be27626ab4d5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:06.914192 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.914164 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bdkgr"] Sep 16 13:18:06.918816 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.918781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:06.935411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.935383 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bdkgr"] Sep 16 13:18:06.998951 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.998906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n477z\" (UniqueName: \"kubernetes.io/projected/8e4d6f37-8a71-4355-937a-be27626ab4d5-kube-api-access-n477z\") pod \"payload-pre-processing-59fbc47bf8-l46wn\" (UID: \"8e4d6f37-8a71-4355-937a-be27626ab4d5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:06.999113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.998963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtphr\" (UniqueName: \"kubernetes.io/projected/82ffd13e-bd27-4e53-b18f-ec8e8845f2a7-kube-api-access-rtphr\") pod \"payload-processing-5b54b7fbb7-bdkgr\" (UID: \"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:06.999113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.999071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ffd13e-bd27-4e53-b18f-ec8e8845f2a7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bdkgr\" (UID: \"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:06.999202 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.999119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4d6f37-8a71-4355-937a-be27626ab4d5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-l46wn\" (UID: \"8e4d6f37-8a71-4355-937a-be27626ab4d5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:06.999920 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:06.999899 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4d6f37-8a71-4355-937a-be27626ab4d5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-l46wn\" (UID: \"8e4d6f37-8a71-4355-937a-be27626ab4d5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:07.010302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.010260 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n477z\" (UniqueName: \"kubernetes.io/projected/8e4d6f37-8a71-4355-937a-be27626ab4d5-kube-api-access-n477z\") pod \"payload-pre-processing-59fbc47bf8-l46wn\" (UID: \"8e4d6f37-8a71-4355-937a-be27626ab4d5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:07.100035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.099999 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtphr\" (UniqueName: \"kubernetes.io/projected/82ffd13e-bd27-4e53-b18f-ec8e8845f2a7-kube-api-access-rtphr\") pod \"payload-processing-5b54b7fbb7-bdkgr\" (UID: \"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:07.100222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.100075 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ffd13e-bd27-4e53-b18f-ec8e8845f2a7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bdkgr\" (UID: \"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:07.100751 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.100730 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ffd13e-bd27-4e53-b18f-ec8e8845f2a7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bdkgr\" (UID: \"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:07.102905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.102890 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:07.119142 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.119118 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtphr\" (UniqueName: \"kubernetes.io/projected/82ffd13e-bd27-4e53-b18f-ec8e8845f2a7-kube-api-access-rtphr\") pod \"payload-processing-5b54b7fbb7-bdkgr\" (UID: \"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:07.229413 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.229378 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:07.387405 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:18:07.387379 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4d6f37_8a71_4355_937a_be27626ab4d5.slice/crio-4e407fdc2da9470df8ec212f40a920f170c139d36e09b494c249fb664c91edf5 WatchSource:0}: Error finding container 4e407fdc2da9470df8ec212f40a920f170c139d36e09b494c249fb664c91edf5: Status 404 returned error can't find the container with id 4e407fdc2da9470df8ec212f40a920f170c139d36e09b494c249fb664c91edf5 Sep 16 13:18:07.387710 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.387689 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn"] Sep 16 13:18:07.512176 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:07.512101 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bdkgr"] Sep 16 13:18:07.514494 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:18:07.514472 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ffd13e_bd27_4e53_b18f_ec8e8845f2a7.slice/crio-6a89c14d87a74eafb21282dd9a1382df29f481b5d36a9f5c13ce9d418aebef3c WatchSource:0}: Error finding container 6a89c14d87a74eafb21282dd9a1382df29f481b5d36a9f5c13ce9d418aebef3c: Status 404 returned error can't find the container with id 6a89c14d87a74eafb21282dd9a1382df29f481b5d36a9f5c13ce9d418aebef3c Sep 16 13:18:08.065443 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:08.065404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" event={"ID":"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7","Type":"ContainerStarted","Data":"6a89c14d87a74eafb21282dd9a1382df29f481b5d36a9f5c13ce9d418aebef3c"} Sep 16 13:18:08.067181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:08.067148 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" event={"ID":"8e4d6f37-8a71-4355-937a-be27626ab4d5","Type":"ContainerStarted","Data":"4e407fdc2da9470df8ec212f40a920f170c139d36e09b494c249fb664c91edf5"} Sep 16 13:18:09.076077 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:09.075991 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" event={"ID":"82ffd13e-bd27-4e53-b18f-ec8e8845f2a7","Type":"ContainerStarted","Data":"11d56a61f57e0bea8650cf3157c527d2f435a94e3ab31a61b891579c0f3a73d2"} Sep 16 13:18:09.076511 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:09.076102 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:09.077761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:09.077736 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" event={"ID":"8e4d6f37-8a71-4355-937a-be27626ab4d5","Type":"ContainerStarted","Data":"988e6fe33d240cb9ab2a75e9f095cba35206d337b6ddbf30c822db072f0e14c3"} Sep 16 13:18:09.077880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:09.077870 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:09.092252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:09.092180 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" podStartSLOduration=1.78502788 podStartE2EDuration="3.09215315s" totalImagesPullingTime="1.30712527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:09.090980868 +0000 UTC m=+574.210707444" watchObservedRunningTime="2026-09-16 13:18:09.09215315 +0000 UTC m=+574.211879703" Sep 16 13:18:09.106812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:09.106770 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" podStartSLOduration=1.674548165 podStartE2EDuration="3.106754401s" totalImagesPullingTime="1.432206236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:09.106649838 +0000 UTC m=+574.226376413" watchObservedRunningTime="2026-09-16 13:18:09.106754401 +0000 UTC m=+574.226480951" Sep 16 13:18:16.968185 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:16.968150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:17.064902 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:17.064864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:19.458475 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:19.458438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:19.557902 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:19.557865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:20.084447 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:20.084419 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-l46wn" Sep 16 13:18:20.084685 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:20.084659 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bdkgr" Sep 16 13:18:28.436553 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:28.436508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:35.347055 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:35.347027 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:18:35.347493 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:35.347033 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:18:37.194648 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.194616 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj"] Sep 16 13:18:37.197936 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.197919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.200295 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.200258 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 13:18:37.200419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.200263 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 13:18:37.201130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.201113 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bdc7l\"" Sep 16 13:18:37.201186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.201147 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 16 13:18:37.206976 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.206955 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj"] Sep 16 13:18:37.253224 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.253194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqx2\" (UniqueName: \"kubernetes.io/projected/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-kube-api-access-njqx2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.253224 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.253232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.253489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.253314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.253489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.253352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.253489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.253411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.253489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.253433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.300030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.299970 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:37.354030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.353997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354156 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354344 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354267 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqx2\" (UniqueName: \"kubernetes.io/projected/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-kube-api-access-njqx2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354344 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354470 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354453 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354561 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.354695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.354680 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.356456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.356439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.356636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.356619 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.364223 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.364202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqx2\" (UniqueName: \"kubernetes.io/projected/66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2-kube-api-access-njqx2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5lkfj\" (UID: \"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.507860 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.507829 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:37.781185 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:37.781112 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj"] Sep 16 13:18:37.784482 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:18:37.784455 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dd56f5_81b4_4f2c_a4cc_e027b4f4e3a2.slice/crio-2c96697b23668fea6947966cbc82e5cc724e149c48133e4ef8fb23e7a593fbf4 WatchSource:0}: Error finding container 2c96697b23668fea6947966cbc82e5cc724e149c48133e4ef8fb23e7a593fbf4: Status 404 returned error can't find the container with id 2c96697b23668fea6947966cbc82e5cc724e149c48133e4ef8fb23e7a593fbf4 Sep 16 13:18:38.235365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:38.235319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" event={"ID":"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2","Type":"ContainerStarted","Data":"2c96697b23668fea6947966cbc82e5cc724e149c48133e4ef8fb23e7a593fbf4"} Sep 16 13:18:39.242596 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:39.242559 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" event={"ID":"66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2","Type":"ContainerStarted","Data":"c3ff115c0e02967ca2704c27ceef5c76f970a76ffb8b9680531ccc91cffe88e8"} Sep 16 13:18:39.261911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:39.261870 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" podStartSLOduration=1.240054578 podStartE2EDuration="2.261850414s" totalImagesPullingTime="1.021795836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:39.260823224 +0000 UTC m=+604.380549772" watchObservedRunningTime="2026-09-16 13:18:39.261850414 +0000 UTC m=+604.381576964" Sep 16 13:18:40.142609 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:40.142569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:40.245070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:40.245039 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:47.508171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:47.508138 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:47.508582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:47.508197 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:47.521039 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:47.521013 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:48.057994 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:48.057962 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:18:48.298988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:48.298960 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5lkfj" Sep 16 13:18:55.515293 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:18:55.515246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:19:05.855796 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:05.855712 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:19:11.484336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.484298 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7"] Sep 16 13:19:11.487965 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.487941 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.490436 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.490408 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 16 13:19:11.498998 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.498973 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7"] Sep 16 13:19:11.590694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.590665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:19:11.625654 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.625624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.625799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.625663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jdb\" (UniqueName: \"kubernetes.io/projected/46a1594a-f60f-4644-9a02-301a020c017a-kube-api-access-26jdb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.625799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.625723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.625799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.625752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.625799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.625782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.625960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.625804 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a1594a-f60f-4644-9a02-301a020c017a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a1594a-f60f-4644-9a02-301a020c017a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jdb\" (UniqueName: \"kubernetes.io/projected/46a1594a-f60f-4644-9a02-301a020c017a-kube-api-access-26jdb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727691 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727764 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.727787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.727773 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.729565 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.729545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a1594a-f60f-4644-9a02-301a020c017a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.729849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.729830 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a1594a-f60f-4644-9a02-301a020c017a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.735118 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.735066 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jdb\" (UniqueName: \"kubernetes.io/projected/46a1594a-f60f-4644-9a02-301a020c017a-kube-api-access-26jdb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8xc7\" (UID: \"46a1594a-f60f-4644-9a02-301a020c017a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:11.798972 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:11.798944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:12.070892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:12.070867 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7"] Sep 16 13:19:12.073460 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:19:12.073431 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a1594a_f60f_4644_9a02_301a020c017a.slice/crio-dc06325ac208e128434440470d0aca4e9ee56f453b22a1b4330789cd9a7eebe7 WatchSource:0}: Error finding container dc06325ac208e128434440470d0aca4e9ee56f453b22a1b4330789cd9a7eebe7: Status 404 returned error can't find the container with id dc06325ac208e128434440470d0aca4e9ee56f453b22a1b4330789cd9a7eebe7 Sep 16 13:19:12.074849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:12.074828 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:19:12.417881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:12.417840 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" event={"ID":"46a1594a-f60f-4644-9a02-301a020c017a","Type":"ContainerStarted","Data":"bacd8deccd95113e5a317e0da8ca78852b4587f919aa3da26a99782d65f3ee60"} Sep 16 13:19:12.418054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:12.417910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" event={"ID":"46a1594a-f60f-4644-9a02-301a020c017a","Type":"ContainerStarted","Data":"dc06325ac208e128434440470d0aca4e9ee56f453b22a1b4330789cd9a7eebe7"} Sep 16 13:19:12.438492 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:12.438446 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" podStartSLOduration=1.229945471 podStartE2EDuration="1.438430316s" totalImagesPullingTime="208.484845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:19:12.436410934 +0000 UTC m=+637.556137509" watchObservedRunningTime="2026-09-16 13:19:12.438430316 +0000 UTC m=+637.558156864" Sep 16 13:19:21.799887 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:21.799844 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:21.800342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:21.799916 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:21.811902 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:21.811882 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:21.870624 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:21.870593 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:19:22.479244 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:22.479207 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8xc7" Sep 16 13:19:31.513609 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:31.513576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:19:39.725914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:19:39.725882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:36.389877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:36.389780 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:36.488811 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:36.488776 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:37.675684 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:37.675643 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:37.772648 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:37.772620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:39.703218 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:39.703182 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:39.808729 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:39.808697 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:58.995997 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:58.995956 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.101212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.101179 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.202095 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.202047 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.391884 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.391783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.420150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.420118 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5"] Sep 16 13:20:59.423694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.423671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.428119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.428101 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a21a40-openshift-default-dockercfg-xt8tb\"" Sep 16 13:20:59.440637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.440614 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5"] Sep 16 13:20:59.505617 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.505581 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.537746 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.537711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-podinfo\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.537903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.537772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22s6\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-kube-api-access-g22s6\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.537903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.537864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-data\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.538018 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.537910 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-credential-socket\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.538018 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.537983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-envoy\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.538018 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.538009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98218b1f-f697-4fbb-960d-bb45b6f38558-istiod-ca-cert\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.538158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.538046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-certs\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.538158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.538071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-socket\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.538158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.538096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-token\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.587672 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.587636 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.638967 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.638934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-token\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-podinfo\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22s6\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-kube-api-access-g22s6\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-data\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-credential-socket\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-envoy\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98218b1f-f697-4fbb-960d-bb45b6f38558-istiod-ca-cert\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-certs\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.639775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.639669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-socket\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.640140 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.640117 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-certs\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.640323 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.640247 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-credential-socket\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.640488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.640394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-socket\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.640657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.640625 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-data\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.640782 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.640761 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98218b1f-f697-4fbb-960d-bb45b6f38558-istiod-ca-cert\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.642269 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.642221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-podinfo\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.642419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.642397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-envoy\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.650850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.650691 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-token\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.650850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.650788 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22s6\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-kube-api-access-g22s6\") pod \"e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.707359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.707327 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:20:59.738385 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.738361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:20:59.809619 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:20:59.809587 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:00.018919 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.018895 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5"] Sep 16 13:21:00.021395 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:00.021368 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98218b1f_f697_4fbb_960d_bb45b6f38558.slice/crio-628be1255928000d612caee390be9dedce24f5a9ce4c3dbf0d47e6511f5f2c37 WatchSource:0}: Error finding container 628be1255928000d612caee390be9dedce24f5a9ce4c3dbf0d47e6511f5f2c37: Status 404 returned error can't find the container with id 628be1255928000d612caee390be9dedce24f5a9ce4c3dbf0d47e6511f5f2c37 Sep 16 13:21:00.023579 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.023545 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:00.023655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.023607 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:00.023655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.023633 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:00.517418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.517381 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:00.607620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.607590 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:00.974593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.974555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:00.978736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.978710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" event={"ID":"98218b1f-f697-4fbb-960d-bb45b6f38558","Type":"ContainerStarted","Data":"bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158"} Sep 16 13:21:00.978870 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:00.978755 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" event={"ID":"98218b1f-f697-4fbb-960d-bb45b6f38558","Type":"ContainerStarted","Data":"628be1255928000d612caee390be9dedce24f5a9ce4c3dbf0d47e6511f5f2c37"} Sep 16 13:21:01.000954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:01.000912 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podStartSLOduration=2.000898641 podStartE2EDuration="2.000898641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:00.999750652 +0000 UTC m=+746.119477230" watchObservedRunningTime="2026-09-16 13:21:01.000898641 +0000 UTC m=+746.120625189" Sep 16 13:21:01.739538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:01.739500 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:01.744135 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:01.744110 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:01.984543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:01.984514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:01.986095 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:01.986080 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:02.812472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:02.812430 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:02.911839 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:02.911805 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:03.011964 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:03.011925 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:03.114315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:03.114195 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:03.212876 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:03.212847 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:04.566426 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:04.566389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:04.656300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:04.656251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:04.796171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:04.796129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:05.153686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.153654 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:05.245318 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.245266 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:05.272890 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.272862 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2"] Sep 16 13:21:05.276238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.276220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.278927 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.278909 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f27ea8\"" Sep 16 13:21:05.290668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.290649 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2"] Sep 16 13:21:05.295052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.295025 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5"] Sep 16 13:21:05.299014 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.298995 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.301393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.301374 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f27ea8-dockercfg-twhgh\"" Sep 16 13:21:05.306949 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.306931 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5"] Sep 16 13:21:05.396504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.396474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f809c5ee-091c-4d14-a5bd-a28baa1d5f41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5\" (UID: \"f809c5ee-091c-4d14-a5bd-a28baa1d5f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.396699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.396531 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwwh\" (UniqueName: \"kubernetes.io/projected/4244d0df-31cb-4fe0-9953-8145ae2a3e88-kube-api-access-fbwwh\") pod \"payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2\" (UID: \"4244d0df-31cb-4fe0-9953-8145ae2a3e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.396699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.396583 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zkg\" (UniqueName: \"kubernetes.io/projected/f809c5ee-091c-4d14-a5bd-a28baa1d5f41-kube-api-access-74zkg\") pod \"payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5\" (UID: \"f809c5ee-091c-4d14-a5bd-a28baa1d5f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.396699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.396603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244d0df-31cb-4fe0-9953-8145ae2a3e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2\" (UID: \"4244d0df-31cb-4fe0-9953-8145ae2a3e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.497964 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.497932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f809c5ee-091c-4d14-a5bd-a28baa1d5f41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5\" (UID: \"f809c5ee-091c-4d14-a5bd-a28baa1d5f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.498145 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.498031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwwh\" (UniqueName: \"kubernetes.io/projected/4244d0df-31cb-4fe0-9953-8145ae2a3e88-kube-api-access-fbwwh\") pod \"payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2\" (UID: \"4244d0df-31cb-4fe0-9953-8145ae2a3e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.498145 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.498071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zkg\" (UniqueName: \"kubernetes.io/projected/f809c5ee-091c-4d14-a5bd-a28baa1d5f41-kube-api-access-74zkg\") pod \"payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5\" (UID: \"f809c5ee-091c-4d14-a5bd-a28baa1d5f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.498145 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.498099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244d0df-31cb-4fe0-9953-8145ae2a3e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2\" (UID: \"4244d0df-31cb-4fe0-9953-8145ae2a3e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.498900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.498872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f809c5ee-091c-4d14-a5bd-a28baa1d5f41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5\" (UID: \"f809c5ee-091c-4d14-a5bd-a28baa1d5f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.498951 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.498925 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4244d0df-31cb-4fe0-9953-8145ae2a3e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2\" (UID: \"4244d0df-31cb-4fe0-9953-8145ae2a3e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.506615 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.506591 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zkg\" (UniqueName: \"kubernetes.io/projected/f809c5ee-091c-4d14-a5bd-a28baa1d5f41-kube-api-access-74zkg\") pod \"payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5\" (UID: \"f809c5ee-091c-4d14-a5bd-a28baa1d5f41\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.506912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.506894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwwh\" (UniqueName: \"kubernetes.io/projected/4244d0df-31cb-4fe0-9953-8145ae2a3e88-kube-api-access-fbwwh\") pod \"payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2\" (UID: \"4244d0df-31cb-4fe0-9953-8145ae2a3e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.586900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.586771 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:05.609923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.609892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:05.634049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.634009 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:05.706108 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.705944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:05.873795 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.873769 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2"] Sep 16 13:21:05.875911 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:05.875885 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4244d0df_31cb_4fe0_9953_8145ae2a3e88.slice/crio-15196f70674b37c73749af4d5d455c235b6e168c9bf7cb38028fc0223ef14297 WatchSource:0}: Error finding container 15196f70674b37c73749af4d5d455c235b6e168c9bf7cb38028fc0223ef14297: Status 404 returned error can't find the container with id 15196f70674b37c73749af4d5d455c235b6e168c9bf7cb38028fc0223ef14297 Sep 16 13:21:05.906315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:05.906268 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5"] Sep 16 13:21:05.907142 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:05.907119 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf809c5ee_091c_4d14_a5bd_a28baa1d5f41.slice/crio-a77c7680905a08443728c4eacf002cc3fbfe10c2dc44c1f501d0091cfe8fa49c WatchSource:0}: Error finding container a77c7680905a08443728c4eacf002cc3fbfe10c2dc44c1f501d0091cfe8fa49c: Status 404 returned error can't find the container with id a77c7680905a08443728c4eacf002cc3fbfe10c2dc44c1f501d0091cfe8fa49c Sep 16 13:21:06.008765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.008726 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" event={"ID":"f809c5ee-091c-4d14-a5bd-a28baa1d5f41","Type":"ContainerStarted","Data":"3515602e166551b8c2266ad2a0e9ba2d764ca6914a150e6e3575758690773a99"} Sep 16 13:21:06.008939 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.008791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" event={"ID":"f809c5ee-091c-4d14-a5bd-a28baa1d5f41","Type":"ContainerStarted","Data":"a77c7680905a08443728c4eacf002cc3fbfe10c2dc44c1f501d0091cfe8fa49c"} Sep 16 13:21:06.009024 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.008998 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:06.010950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.010914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" event={"ID":"4244d0df-31cb-4fe0-9953-8145ae2a3e88","Type":"ContainerStarted","Data":"5f3b15d8412ea6674616ba5e8fc8d0d574ddfae0f8ba4d87f8b0280f4ff267c3"} Sep 16 13:21:06.011093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.010967 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:06.011093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.010996 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" event={"ID":"4244d0df-31cb-4fe0-9953-8145ae2a3e88","Type":"ContainerStarted","Data":"15196f70674b37c73749af4d5d455c235b6e168c9bf7cb38028fc0223ef14297"} Sep 16 13:21:06.029416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.029372 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" podStartSLOduration=1.029356575 podStartE2EDuration="1.029356575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:06.026540313 +0000 UTC m=+751.146266886" watchObservedRunningTime="2026-09-16 13:21:06.029356575 +0000 UTC m=+751.149083124" Sep 16 13:21:06.031805 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.031769 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:06.050722 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.050685 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" podStartSLOduration=1.050672537 podStartE2EDuration="1.050672537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:06.048828233 +0000 UTC m=+751.168554811" watchObservedRunningTime="2026-09-16 13:21:06.050672537 +0000 UTC m=+751.170399086" Sep 16 13:21:06.189334 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.189222 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:06.244002 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.243966 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6"] Sep 16 13:21:06.247612 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.247586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.249773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.249754 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-48adf5\"" Sep 16 13:21:06.249900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.249780 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-48adf5-dockercfg-f2rmf\"" Sep 16 13:21:06.257824 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.257800 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6"] Sep 16 13:21:06.281238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.281202 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:06.281457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.281437 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd"] Sep 16 13:21:06.284871 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.284847 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.296606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.296575 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd"] Sep 16 13:21:06.407555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.407511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e1be0-8ff6-466e-9e94-68904cb8290a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd\" (UID: \"238e1be0-8ff6-466e-9e94-68904cb8290a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.407722 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.407562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8268d4f-2d67-4f3c-b757-61161d1c2b2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6\" (UID: \"b8268d4f-2d67-4f3c-b757-61161d1c2b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.407722 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.407642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklxd\" (UniqueName: \"kubernetes.io/projected/238e1be0-8ff6-466e-9e94-68904cb8290a-kube-api-access-lklxd\") pod \"payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd\" (UID: \"238e1be0-8ff6-466e-9e94-68904cb8290a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.407821 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.407733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkpq\" (UniqueName: \"kubernetes.io/projected/b8268d4f-2d67-4f3c-b757-61161d1c2b2d-kube-api-access-hfkpq\") pod \"payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6\" (UID: \"b8268d4f-2d67-4f3c-b757-61161d1c2b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.508655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.508624 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e1be0-8ff6-466e-9e94-68904cb8290a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd\" (UID: \"238e1be0-8ff6-466e-9e94-68904cb8290a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.508826 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.508691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8268d4f-2d67-4f3c-b757-61161d1c2b2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6\" (UID: \"b8268d4f-2d67-4f3c-b757-61161d1c2b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.508826 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.508733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklxd\" (UniqueName: \"kubernetes.io/projected/238e1be0-8ff6-466e-9e94-68904cb8290a-kube-api-access-lklxd\") pod \"payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd\" (UID: \"238e1be0-8ff6-466e-9e94-68904cb8290a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.508826 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.508815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkpq\" (UniqueName: \"kubernetes.io/projected/b8268d4f-2d67-4f3c-b757-61161d1c2b2d-kube-api-access-hfkpq\") pod \"payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6\" (UID: \"b8268d4f-2d67-4f3c-b757-61161d1c2b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.509485 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.509461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/238e1be0-8ff6-466e-9e94-68904cb8290a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd\" (UID: \"238e1be0-8ff6-466e-9e94-68904cb8290a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.509636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.509616 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8268d4f-2d67-4f3c-b757-61161d1c2b2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6\" (UID: \"b8268d4f-2d67-4f3c-b757-61161d1c2b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.517186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.517168 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklxd\" (UniqueName: \"kubernetes.io/projected/238e1be0-8ff6-466e-9e94-68904cb8290a-kube-api-access-lklxd\") pod \"payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd\" (UID: \"238e1be0-8ff6-466e-9e94-68904cb8290a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.517418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.517397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkpq\" (UniqueName: \"kubernetes.io/projected/b8268d4f-2d67-4f3c-b757-61161d1c2b2d-kube-api-access-hfkpq\") pod \"payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6\" (UID: \"b8268d4f-2d67-4f3c-b757-61161d1c2b2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.559817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.559791 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:06.600712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.600683 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:06.731956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.731859 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:06.822364 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.822266 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:06.846220 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.846193 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6"] Sep 16 13:21:06.847715 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:06.847682 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8268d4f_2d67_4f3c_b757_61161d1c2b2d.slice/crio-15a3684428d25df79a83d6cccdcc41c87bc5c80024007fd457bcb7f7c8c4a2af WatchSource:0}: Error finding container 15a3684428d25df79a83d6cccdcc41c87bc5c80024007fd457bcb7f7c8c4a2af: Status 404 returned error can't find the container with id 15a3684428d25df79a83d6cccdcc41c87bc5c80024007fd457bcb7f7c8c4a2af Sep 16 13:21:06.896637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.896614 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd"] Sep 16 13:21:06.899017 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:06.898989 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238e1be0_8ff6_466e_9e94_68904cb8290a.slice/crio-b71c56f8a8cc273a1509c268bfddbccb0a512c078c522c66c88163b966d37912 WatchSource:0}: Error finding container b71c56f8a8cc273a1509c268bfddbccb0a512c078c522c66c88163b966d37912: Status 404 returned error can't find the container with id b71c56f8a8cc273a1509c268bfddbccb0a512c078c522c66c88163b966d37912 Sep 16 13:21:06.926204 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:06.926171 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.023093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.023050 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" event={"ID":"238e1be0-8ff6-466e-9e94-68904cb8290a","Type":"ContainerStarted","Data":"3a02f68ca4e0de8e6eaa7057549109e63b404f71aa303def7334ba569ddc1e7e"} Sep 16 13:21:07.023306 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.023126 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" event={"ID":"238e1be0-8ff6-466e-9e94-68904cb8290a","Type":"ContainerStarted","Data":"b71c56f8a8cc273a1509c268bfddbccb0a512c078c522c66c88163b966d37912"} Sep 16 13:21:07.023484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.023454 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:07.025829 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.025801 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" event={"ID":"b8268d4f-2d67-4f3c-b757-61161d1c2b2d","Type":"ContainerStarted","Data":"03319fc0113ece5a6dc8ca39aa32f062b4078e2c99694fc9e74b7aca46046953"} Sep 16 13:21:07.025959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.025846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" event={"ID":"b8268d4f-2d67-4f3c-b757-61161d1c2b2d","Type":"ContainerStarted","Data":"15a3684428d25df79a83d6cccdcc41c87bc5c80024007fd457bcb7f7c8c4a2af"} Sep 16 13:21:07.025959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.025894 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:07.049387 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.049328 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" podStartSLOduration=1.049266664 podStartE2EDuration="1.049266664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:07.04611597 +0000 UTC m=+752.165842541" watchObservedRunningTime="2026-09-16 13:21:07.049266664 +0000 UTC m=+752.168993250" Sep 16 13:21:07.066368 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.066322 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" podStartSLOduration=1.066305972 podStartE2EDuration="1.066305972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:07.063563157 +0000 UTC m=+752.183289730" watchObservedRunningTime="2026-09-16 13:21:07.066305972 +0000 UTC m=+752.186032521" Sep 16 13:21:07.138663 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.138565 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.492604 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.492571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.580222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.580191 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.654130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.654100 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl"] Sep 16 13:21:07.658023 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.658003 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.660557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.660523 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a21a40\"" Sep 16 13:21:07.660671 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.660602 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a21a40-dockercfg-fbnqs\"" Sep 16 13:21:07.669424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.669402 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl"] Sep 16 13:21:07.669669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.669649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:07.824709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.824615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndg7\" (UniqueName: \"kubernetes.io/projected/14c496d8-d00a-42b5-9e18-35d42904f774-kube-api-access-tndg7\") pod \"payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl\" (UID: \"14c496d8-d00a-42b5-9e18-35d42904f774\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.824853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.824739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c496d8-d00a-42b5-9e18-35d42904f774-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl\" (UID: \"14c496d8-d00a-42b5-9e18-35d42904f774\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.925985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.925952 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c496d8-d00a-42b5-9e18-35d42904f774-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl\" (UID: \"14c496d8-d00a-42b5-9e18-35d42904f774\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.926172 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.926016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndg7\" (UniqueName: \"kubernetes.io/projected/14c496d8-d00a-42b5-9e18-35d42904f774-kube-api-access-tndg7\") pod \"payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl\" (UID: \"14c496d8-d00a-42b5-9e18-35d42904f774\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.926895 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.926872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c496d8-d00a-42b5-9e18-35d42904f774-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl\" (UID: \"14c496d8-d00a-42b5-9e18-35d42904f774\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.935427 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.935397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndg7\" (UniqueName: \"kubernetes.io/projected/14c496d8-d00a-42b5-9e18-35d42904f774-kube-api-access-tndg7\") pod \"payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl\" (UID: \"14c496d8-d00a-42b5-9e18-35d42904f774\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:07.970078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:07.970048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:08.044419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:08.044367 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:08.248573 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:08.248546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl"] Sep 16 13:21:08.249650 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:08.249625 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c496d8_d00a_42b5_9e18_35d42904f774.slice/crio-2f42f257f1b132241864f29e45d5391049142e61599c858bd22d551f44cdec71 WatchSource:0}: Error finding container 2f42f257f1b132241864f29e45d5391049142e61599c858bd22d551f44cdec71: Status 404 returned error can't find the container with id 2f42f257f1b132241864f29e45d5391049142e61599c858bd22d551f44cdec71 Sep 16 13:21:08.652521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:08.652444 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:08.753362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:08.753326 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:08.903380 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:08.903259 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:09.004798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.004768 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:09.042644 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.042614 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" event={"ID":"14c496d8-d00a-42b5-9e18-35d42904f774","Type":"ContainerStarted","Data":"28e1e56f231565bfc2b1a2be89b1eaf5bd2f23bdb1704cddf010261b3d6f1247"} Sep 16 13:21:09.042819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.042678 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" event={"ID":"14c496d8-d00a-42b5-9e18-35d42904f774","Type":"ContainerStarted","Data":"2f42f257f1b132241864f29e45d5391049142e61599c858bd22d551f44cdec71"} Sep 16 13:21:09.042819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.042709 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:09.060093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.060046 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" podStartSLOduration=2.060033152 podStartE2EDuration="2.060033152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:09.058519742 +0000 UTC m=+754.178246310" watchObservedRunningTime="2026-09-16 13:21:09.060033152 +0000 UTC m=+754.179759700" Sep 16 13:21:09.198496 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.198411 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn"] Sep 16 13:21:09.204968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.202752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.208698 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.207125 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn"] Sep 16 13:21:09.208698 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.208051 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ef119741-openshift-default-dockercfg-9mtgq\"" Sep 16 13:21:09.225092 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.225060 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:09.339812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.339778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-envoy\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.339822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhvz\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-kube-api-access-xdhvz\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.339886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-istio-token\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.339935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-certs\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340026 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.340010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215fce8d-a522-4be2-b130-fe67a679d22d-istio-podinfo\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340261 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.340033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215fce8d-a522-4be2-b130-fe67a679d22d-istiod-ca-cert\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340261 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.340058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-credential-socket\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340261 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.340139 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-data\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.340261 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.340197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-socket\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.440681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215fce8d-a522-4be2-b130-fe67a679d22d-istio-podinfo\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.440681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215fce8d-a522-4be2-b130-fe67a679d22d-istiod-ca-cert\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.440915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-credential-socket\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.440915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-data\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.440915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-socket\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-envoy\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.440975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhvz\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-kube-api-access-xdhvz\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-istio-token\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441032 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-credential-socket\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-certs\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441158 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-data\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441176 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-socket\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441419 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-certs\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.441511 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.441492 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215fce8d-a522-4be2-b130-fe67a679d22d-istiod-ca-cert\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.443236 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.443205 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-envoy\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.443376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.443358 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215fce8d-a522-4be2-b130-fe67a679d22d-istio-podinfo\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.454228 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.454200 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-istio-token\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.455156 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.455131 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhvz\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-kube-api-access-xdhvz\") pod \"e2e-ait-ef119741-openshift-default-7c785c776-9hpnn\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.483212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.483186 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:09.517589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.517564 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:09.795360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.795264 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn"] Sep 16 13:21:09.796943 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:09.796906 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215fce8d_a522_4be2_b130_fe67a679d22d.slice/crio-0d6588a18f6f929a521923e237217a8de3605f5f0d7f36c6eb91102fc8646cd1 WatchSource:0}: Error finding container 0d6588a18f6f929a521923e237217a8de3605f5f0d7f36c6eb91102fc8646cd1: Status 404 returned error can't find the container with id 0d6588a18f6f929a521923e237217a8de3605f5f0d7f36c6eb91102fc8646cd1 Sep 16 13:21:09.800144 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.800101 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:09.800242 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.800181 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:09.800335 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.800249 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:09.800512 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:09.800415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:10.057789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.057687 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" event={"ID":"215fce8d-a522-4be2-b130-fe67a679d22d","Type":"ContainerStarted","Data":"ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0"} Sep 16 13:21:10.057789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.057780 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" event={"ID":"215fce8d-a522-4be2-b130-fe67a679d22d","Type":"ContainerStarted","Data":"0d6588a18f6f929a521923e237217a8de3605f5f0d7f36c6eb91102fc8646cd1"} Sep 16 13:21:10.079749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.079464 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podStartSLOduration=1.079444822 podStartE2EDuration="1.079444822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:10.0786681 +0000 UTC m=+755.198394671" watchObservedRunningTime="2026-09-16 13:21:10.079444822 +0000 UTC m=+755.199171373" Sep 16 13:21:10.517908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.517878 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:10.522872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.522847 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:10.833726 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.833637 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:10.984922 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:10.984879 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:11.063191 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:11.063155 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:11.064065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:11.064045 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:11.089569 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:11.089481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:11.540416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:11.540376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:11.713961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:11.713918 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:12.580271 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:12.580244 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:13.672023 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:13.671995 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:15.002300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:14.992909 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:15.090959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:15.090899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:15.426890 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:15.426810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:15.754766 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:15.754736 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:15.849594 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:15.849562 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:16.249181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:16.249148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:17.028516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.028477 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5" Sep 16 13:21:17.028748 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.028726 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2" Sep 16 13:21:17.343308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.343200 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:17.451951 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.451914 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:17.552350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.552311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:17.667484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.667458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:17.767123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.767090 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:17.874616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:17.874579 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.008466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.008414 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.038360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.038332 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6" Sep 16 13:21:18.038544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.038410 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd" Sep 16 13:21:18.115699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.115665 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq"] Sep 16 13:21:18.120260 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.120235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.123346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.123322 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-15da7b\"" Sep 16 13:21:18.128123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.128094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-15da7b-dockercfg-d9n4d\"" Sep 16 13:21:18.135179 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.135154 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq"] Sep 16 13:21:18.178161 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.178088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.189529 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.189499 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng"] Sep 16 13:21:18.193616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.193593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.208266 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.208241 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng"] Sep 16 13:21:18.234892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.234867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.235022 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.234942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmnb\" (UniqueName: \"kubernetes.io/projected/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-kube-api-access-pjmnb\") pod \"payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.311064 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.310915 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.336345 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.336277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.336500 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.336406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccbedcb-9793-4447-a09b-89ff6c1e6179-plugins-config-volume\") pod \"payload-processing-e2e-autores-15da7b-688d5f8989-58lng\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.336500 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.336443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmnb\" (UniqueName: \"kubernetes.io/projected/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-kube-api-access-pjmnb\") pod \"payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.336500 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.336477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9848m\" (UniqueName: \"kubernetes.io/projected/fccbedcb-9793-4447-a09b-89ff6c1e6179-kube-api-access-9848m\") pod \"payload-processing-e2e-autores-15da7b-688d5f8989-58lng\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.337119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.337100 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.360967 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.360939 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmnb\" (UniqueName: \"kubernetes.io/projected/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-kube-api-access-pjmnb\") pod \"payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.403393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.403363 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.434052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.434022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:18.438412 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.437420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccbedcb-9793-4447-a09b-89ff6c1e6179-plugins-config-volume\") pod \"payload-processing-e2e-autores-15da7b-688d5f8989-58lng\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.438412 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.437469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9848m\" (UniqueName: \"kubernetes.io/projected/fccbedcb-9793-4447-a09b-89ff6c1e6179-kube-api-access-9848m\") pod \"payload-processing-e2e-autores-15da7b-688d5f8989-58lng\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.439487 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.439457 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccbedcb-9793-4447-a09b-89ff6c1e6179-plugins-config-volume\") pod \"payload-processing-e2e-autores-15da7b-688d5f8989-58lng\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.455315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.453850 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9848m\" (UniqueName: \"kubernetes.io/projected/fccbedcb-9793-4447-a09b-89ff6c1e6179-kube-api-access-9848m\") pod \"payload-processing-e2e-autores-15da7b-688d5f8989-58lng\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.504938 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.504736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:18.568943 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.568812 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.666517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.666483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:18.739977 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.739820 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq"] Sep 16 13:21:18.743414 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:18.743380 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbcbd82_0d67_4a36_b089_964b63c3a2d2.slice/crio-96f68b9ae74c2725738350c2082145df427c278a285b30b70f49f6b7d2fc1e51 WatchSource:0}: Error finding container 96f68b9ae74c2725738350c2082145df427c278a285b30b70f49f6b7d2fc1e51: Status 404 returned error can't find the container with id 96f68b9ae74c2725738350c2082145df427c278a285b30b70f49f6b7d2fc1e51 Sep 16 13:21:18.806497 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:18.806472 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng"] Sep 16 13:21:18.807624 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:18.807596 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccbedcb_9793_4447_a09b_89ff6c1e6179.slice/crio-822d986406a2c674432f27cf64be7d883d3397e82b19922d827388e2b4d64e58 WatchSource:0}: Error finding container 822d986406a2c674432f27cf64be7d883d3397e82b19922d827388e2b4d64e58: Status 404 returned error can't find the container with id 822d986406a2c674432f27cf64be7d883d3397e82b19922d827388e2b4d64e58 Sep 16 13:21:19.126784 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.126690 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" event={"ID":"fccbedcb-9793-4447-a09b-89ff6c1e6179","Type":"ContainerStarted","Data":"177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5"} Sep 16 13:21:19.126784 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.126753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" event={"ID":"fccbedcb-9793-4447-a09b-89ff6c1e6179","Type":"ContainerStarted","Data":"822d986406a2c674432f27cf64be7d883d3397e82b19922d827388e2b4d64e58"} Sep 16 13:21:19.127001 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.126875 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:19.128668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.128635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" event={"ID":"4cbcbd82-0d67-4a36-b089-964b63c3a2d2","Type":"ContainerStarted","Data":"b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f"} Sep 16 13:21:19.128820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.128693 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:19.128820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.128717 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" event={"ID":"4cbcbd82-0d67-4a36-b089-964b63c3a2d2","Type":"ContainerStarted","Data":"96f68b9ae74c2725738350c2082145df427c278a285b30b70f49f6b7d2fc1e51"} Sep 16 13:21:19.157250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.157193 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" podStartSLOduration=1.157176451 podStartE2EDuration="1.157176451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:19.155317951 +0000 UTC m=+764.275044530" watchObservedRunningTime="2026-09-16 13:21:19.157176451 +0000 UTC m=+764.276903002" Sep 16 13:21:19.177298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.177237 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" podStartSLOduration=1.177219059 podStartE2EDuration="1.177219059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:19.176076652 +0000 UTC m=+764.295803213" watchObservedRunningTime="2026-09-16 13:21:19.177219059 +0000 UTC m=+764.296945611" Sep 16 13:21:19.609130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.609091 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:19.763796 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.763759 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:19.862777 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.862690 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:19.968664 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:19.968632 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:20.060737 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:20.060707 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl" Sep 16 13:21:20.076336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:20.076307 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:20.649015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:20.648973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:21.142903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.142822 2581 generic.go:356] "Generic (PLEG): container finished" podID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerID="bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158" exitCode=0 Sep 16 13:21:21.142903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.142896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" event={"ID":"98218b1f-f697-4fbb-960d-bb45b6f38558","Type":"ContainerDied","Data":"bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158"} Sep 16 13:21:21.143347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.143313 2581 scope.go:117] "RemoveContainer" containerID="bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158" Sep 16 13:21:21.144403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.144374 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:21.144505 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.144440 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:21.144505 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.144493 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:21.635946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.635910 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:21.851481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.851445 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:21.957749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:21.957716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:22.153511 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:22.153475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" event={"ID":"98218b1f-f697-4fbb-960d-bb45b6f38558","Type":"ContainerStarted","Data":"7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2"} Sep 16 13:21:22.638456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:22.638424 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:22.738574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:22.738544 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:22.738761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:22.738657 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:21:22.738761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:22.738719 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:22.738889 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:22.738717 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 16 13:21:23.492036 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:23.491948 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:23.739381 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:23.739349 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:21:23.739554 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:23.739399 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 16 13:21:24.431425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:24.431387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:24.529717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:24.529688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:24.629961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:24.629921 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:24.739451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:24.739415 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:21:24.739620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:24.739495 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 16 13:21:25.235414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:25.235383 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:25.331061 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:25.331026 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:25.739062 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:25.739028 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:21:25.739475 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:25.739089 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 16 13:21:25.874345 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:25.874301 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:26.739074 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:26.739037 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:21:26.739499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:26.739120 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 16 13:21:27.022820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:27.022696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:27.882207 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:27.882177 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:27.882745 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:27.882626 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:27.890086 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:27.890063 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:21:28.030976 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:28.030933 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:29.060131 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:29.060085 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:30.039532 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:30.039495 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:30.138215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:30.138185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:21:30.138775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:30.138343 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:21:35.256005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.255972 2581 generic.go:356] "Generic (PLEG): container finished" podID="215fce8d-a522-4be2-b130-fe67a679d22d" containerID="ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0" exitCode=0 Sep 16 13:21:35.256414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.256042 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" event={"ID":"215fce8d-a522-4be2-b130-fe67a679d22d","Type":"ContainerDied","Data":"ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0"} Sep 16 13:21:35.256459 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.256430 2581 scope.go:117] "RemoveContainer" containerID="ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0" Sep 16 13:21:35.258749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.258110 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:35.258749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.258300 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:35.258749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.258407 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:21:35.818470 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:35.818414 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:36.231072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:36.231032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:36.263642 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:36.263610 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" event={"ID":"215fce8d-a522-4be2-b130-fe67a679d22d","Type":"ContainerStarted","Data":"642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c"} Sep 16 13:21:36.518582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:36.518494 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:36.519824 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:36.519794 2581 patch_prober.go:28] interesting pod/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 16 13:21:36.519956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:36.519858 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 16 13:21:37.518581 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:37.518544 2581 patch_prober.go:28] interesting pod/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 16 13:21:37.518979 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:37.518599 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 16 13:21:37.945827 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:37.945739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:38.517883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:38.517851 2581 patch_prober.go:28] interesting pod/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 16 13:21:38.518068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:38.517905 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 16 13:21:38.847681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:38.847595 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:39.518651 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:39.518614 2581 patch_prober.go:28] interesting pod/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 16 13:21:39.518834 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:39.518675 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 16 13:21:40.033484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:40.033454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:40.518636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:40.518599 2581 patch_prober.go:28] interesting pod/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 16 13:21:40.518821 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:40.518657 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 16 13:21:40.974613 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:40.974578 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:41.063642 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:41.063607 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:41.599481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:41.599443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:41.600979 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:41.600942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:41.746519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:41.746484 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:42.171550 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.171517 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl"] Sep 16 13:21:42.176473 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.176456 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.178855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.178836 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:21:42.179879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.179850 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:21:42.179879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.179869 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:21:42.183704 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.183628 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl"] Sep 16 13:21:42.277182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.277151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpll\" (UniqueName: \"kubernetes.io/projected/91e507b9-2d02-4415-aa86-7166d086f08a-kube-api-access-4gpll\") pod \"maas-api-revoke-keys-e2e-ait-ef119741-7qwcl\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.277436 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.277210 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e507b9-2d02-4415-aa86-7166d086f08a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef119741-7qwcl\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.378014 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.377984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e507b9-2d02-4415-aa86-7166d086f08a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef119741-7qwcl\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.378188 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.378116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpll\" (UniqueName: \"kubernetes.io/projected/91e507b9-2d02-4415-aa86-7166d086f08a-kube-api-access-4gpll\") pod \"maas-api-revoke-keys-e2e-ait-ef119741-7qwcl\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.378718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.378698 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e507b9-2d02-4415-aa86-7166d086f08a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef119741-7qwcl\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.386376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.386354 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpll\" (UniqueName: \"kubernetes.io/projected/91e507b9-2d02-4415-aa86-7166d086f08a-kube-api-access-4gpll\") pod \"maas-api-revoke-keys-e2e-ait-ef119741-7qwcl\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.492586 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.488459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:42.771330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.771305 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl"] Sep 16 13:21:42.773066 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:21:42.773036 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e507b9_2d02_4415_aa86_7166d086f08a.slice/crio-fdf9bcc54c5832f7765463919d7eb8f16f2439f4e67a2d7b4edc99a4b7c86398 WatchSource:0}: Error finding container fdf9bcc54c5832f7765463919d7eb8f16f2439f4e67a2d7b4edc99a4b7c86398: Status 404 returned error can't find the container with id fdf9bcc54c5832f7765463919d7eb8f16f2439f4e67a2d7b4edc99a4b7c86398 Sep 16 13:21:42.843421 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:42.843386 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:43.307113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:43.307081 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" event={"ID":"91e507b9-2d02-4415-aa86-7166d086f08a","Type":"ContainerStarted","Data":"fdf9bcc54c5832f7765463919d7eb8f16f2439f4e67a2d7b4edc99a4b7c86398"} Sep 16 13:21:43.798339 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:43.798300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:44.517851 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:44.517817 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:45.303405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:45.303373 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:45.320482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:45.320448 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" event={"ID":"91e507b9-2d02-4415-aa86-7166d086f08a","Type":"ContainerStarted","Data":"6e2437ac676e5805898e1ac6c8be58e1addcc3436d941f16756296ab18eafe5d"} Sep 16 13:21:45.340618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:45.340571 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" podStartSLOduration=0.989602019 podStartE2EDuration="3.340556613s" totalImagesPullingTime="2.350954594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:45.338961156 +0000 UTC m=+790.458687746" watchObservedRunningTime="2026-09-16 13:21:45.340556613 +0000 UTC m=+790.460283161" Sep 16 13:21:46.085535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:46.085496 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:46.282031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:46.281994 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:46.327560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:46.327526 2581 generic.go:356] "Generic (PLEG): container finished" podID="91e507b9-2d02-4415-aa86-7166d086f08a" containerID="6e2437ac676e5805898e1ac6c8be58e1addcc3436d941f16756296ab18eafe5d" exitCode=0 Sep 16 13:21:46.327732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:46.327649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" event={"ID":"91e507b9-2d02-4415-aa86-7166d086f08a","Type":"ContainerDied","Data":"6e2437ac676e5805898e1ac6c8be58e1addcc3436d941f16756296ab18eafe5d"} Sep 16 13:21:46.849867 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:46.849831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:47.013473 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.013442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:47.479495 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.479470 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:47.629888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.629808 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gpll\" (UniqueName: \"kubernetes.io/projected/91e507b9-2d02-4415-aa86-7166d086f08a-kube-api-access-4gpll\") pod \"91e507b9-2d02-4415-aa86-7166d086f08a\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " Sep 16 13:21:47.629888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.629852 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e507b9-2d02-4415-aa86-7166d086f08a-maas-api-service-ca\") pod \"91e507b9-2d02-4415-aa86-7166d086f08a\" (UID: \"91e507b9-2d02-4415-aa86-7166d086f08a\") " Sep 16 13:21:47.630266 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.630244 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e507b9-2d02-4415-aa86-7166d086f08a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91e507b9-2d02-4415-aa86-7166d086f08a" (UID: "91e507b9-2d02-4415-aa86-7166d086f08a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:21:47.632174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.632152 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e507b9-2d02-4415-aa86-7166d086f08a-kube-api-access-4gpll" (OuterVolumeSpecName: "kube-api-access-4gpll") pod "91e507b9-2d02-4415-aa86-7166d086f08a" (UID: "91e507b9-2d02-4415-aa86-7166d086f08a"). InnerVolumeSpecName "kube-api-access-4gpll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:21:47.730668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.730636 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e507b9-2d02-4415-aa86-7166d086f08a-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:47.730668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.730665 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpll\" (UniqueName: \"kubernetes.io/projected/91e507b9-2d02-4415-aa86-7166d086f08a-kube-api-access-4gpll\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:47.823943 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:47.823907 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:48.340346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:48.340319 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" Sep 16 13:21:48.340522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:48.340316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl" event={"ID":"91e507b9-2d02-4415-aa86-7166d086f08a","Type":"ContainerDied","Data":"fdf9bcc54c5832f7765463919d7eb8f16f2439f4e67a2d7b4edc99a4b7c86398"} Sep 16 13:21:48.340522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:48.340444 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf9bcc54c5832f7765463919d7eb8f16f2439f4e67a2d7b4edc99a4b7c86398" Sep 16 13:21:48.651598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:48.651508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:49.430257 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:49.430211 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:53.648086 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:53.648051 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:53.747865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:53.747833 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:53.851525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:53.851483 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl"] Sep 16 13:21:53.852775 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:53.852750 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:53.854212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:53.854190 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef119741-7qwcl"] Sep 16 13:21:53.916614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:53.916529 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:54.002275 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.002238 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn"] Sep 16 13:21:54.002570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.002534 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" containerID="cri-o://642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c" gracePeriod=30 Sep 16 13:21:54.031267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.031235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:54.331249 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.331206 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:54.518311 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.518239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:54.617211 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.617134 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:54.728846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:54.728808 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:55.062456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:55.062420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:55.249052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:55.249015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:55.389305 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:55.389205 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e507b9-2d02-4415-aa86-7166d086f08a" path="/var/lib/kubelet/pods/91e507b9-2d02-4415-aa86-7166d086f08a/volumes" Sep 16 13:21:55.523860 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:55.523817 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:56.146506 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:56.146465 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:56.602194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:56.602159 2581 patch_prober.go:28] interesting pod/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:21:56.602379 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:56.602220 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:21:57.000966 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:57.000917 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:57.714453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:57.714412 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:57.815899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:57.815845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:58.525749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:58.525712 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:58.908990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:58.908865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:59.253035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.253009 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:59.335354 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335325 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215fce8d-a522-4be2-b130-fe67a679d22d-istio-podinfo\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.335527 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335386 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-credential-socket\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.335527 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335419 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-istio-token\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.335527 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335439 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdhvz\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-kube-api-access-xdhvz\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335713 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335770 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-certs\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335843 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-envoy\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335869 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-socket\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335899 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215fce8d-a522-4be2-b130-fe67a679d22d-istiod-ca-cert\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335925 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.335935 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-data\") pod \"215fce8d-a522-4be2-b130-fe67a679d22d\" (UID: \"215fce8d-a522-4be2-b130-fe67a679d22d\") " Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336125 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-data" (OuterVolumeSpecName: "istio-data") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336144 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336341 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336363 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336376 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336389 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.336437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.336409 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215fce8d-a522-4be2-b130-fe67a679d22d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:21:59.338127 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.338098 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/215fce8d-a522-4be2-b130-fe67a679d22d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:21:59.338361 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.338335 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-kube-api-access-xdhvz" (OuterVolumeSpecName: "kube-api-access-xdhvz") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "kube-api-access-xdhvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:21:59.338590 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.338569 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-istio-token" (OuterVolumeSpecName: "istio-token") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:21:59.338734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.338717 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "215fce8d-a522-4be2-b130-fe67a679d22d" (UID: "215fce8d-a522-4be2-b130-fe67a679d22d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:21:59.342890 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.342860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:59.403229 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.403198 2581 generic.go:356] "Generic (PLEG): container finished" podID="215fce8d-a522-4be2-b130-fe67a679d22d" containerID="642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c" exitCode=0 Sep 16 13:21:59.403347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.403237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" event={"ID":"215fce8d-a522-4be2-b130-fe67a679d22d","Type":"ContainerDied","Data":"642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c"} Sep 16 13:21:59.403347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.403252 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" Sep 16 13:21:59.403347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.403277 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn" event={"ID":"215fce8d-a522-4be2-b130-fe67a679d22d","Type":"ContainerDied","Data":"0d6588a18f6f929a521923e237217a8de3605f5f0d7f36c6eb91102fc8646cd1"} Sep 16 13:21:59.403347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.403329 2581 scope.go:117] "RemoveContainer" containerID="642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c" Sep 16 13:21:59.413581 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.413556 2581 scope.go:117] "RemoveContainer" containerID="ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0" Sep 16 13:21:59.426717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.426605 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:59.426717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.426631 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn"] Sep 16 13:21:59.433342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.433315 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ef119741-openshift-default-7c785c776-9hpnn"] Sep 16 13:21:59.433674 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.433655 2581 scope.go:117] "RemoveContainer" containerID="642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c" Sep 16 13:21:59.433981 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:21:59.433963 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c\": container with ID starting with 642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c not found: ID does not exist" containerID="642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c" Sep 16 13:21:59.434042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.433992 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c"} err="failed to get container status \"642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c\": rpc error: code = NotFound desc = could not find container \"642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c\": container with ID starting with 642807fc0e3b23e19d47308974d3a57f48d67b97d7545312ae7a009ba9ec735c not found: ID does not exist" Sep 16 13:21:59.434042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.434009 2581 scope.go:117] "RemoveContainer" containerID="ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0" Sep 16 13:21:59.434348 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:21:59.434262 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0\": container with ID starting with ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0 not found: ID does not exist" containerID="ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0" Sep 16 13:21:59.434348 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.434311 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0"} err="failed to get container status \"ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0\": rpc error: code = NotFound desc = could not find container \"ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0\": container with ID starting with ea7e953380e01599cbabdca3cc652916f1dcf1cc2cd5994b368b444b3cbbd8b0 not found: ID does not exist" Sep 16 13:21:59.437931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.437912 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215fce8d-a522-4be2-b130-fe67a679d22d-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.437996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.437936 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.437996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.437951 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdhvz\" (UniqueName: \"kubernetes.io/projected/215fce8d-a522-4be2-b130-fe67a679d22d-kube-api-access-xdhvz\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.437996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.437964 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215fce8d-a522-4be2-b130-fe67a679d22d-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.437996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.437977 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215fce8d-a522-4be2-b130-fe67a679d22d-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:21:59.679408 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.679318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:21:59.939186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:21:59.939099 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:00.027776 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.027746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:00.028305 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028273 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg"] Sep 16 13:22:00.028719 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028706 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" Sep 16 13:22:00.028763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028721 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" Sep 16 13:22:00.028763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028739 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e507b9-2d02-4415-aa86-7166d086f08a" containerName="revoke-keys" Sep 16 13:22:00.028763 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028747 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e507b9-2d02-4415-aa86-7166d086f08a" containerName="revoke-keys" Sep 16 13:22:00.028855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028826 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" Sep 16 13:22:00.028855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.028838 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e507b9-2d02-4415-aa86-7166d086f08a" containerName="revoke-keys" Sep 16 13:22:00.033276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.033260 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.035962 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.035946 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-508c5fc8\"" Sep 16 13:22:00.043552 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.043529 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg"] Sep 16 13:22:00.143846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.143793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgcn\" (UniqueName: \"kubernetes.io/projected/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-kube-api-access-5jgcn\") pod \"payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.144015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.143933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.244770 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.244738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgcn\" (UniqueName: \"kubernetes.io/projected/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-kube-api-access-5jgcn\") pod \"payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.244974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.244811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.245560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.245539 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.253944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.253914 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgcn\" (UniqueName: \"kubernetes.io/projected/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-kube-api-access-5jgcn\") pod \"payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.345413 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.345375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:00.362424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.362398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:00.622147 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:00.622119 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg"] Sep 16 13:22:00.624407 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:22:00.624380 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d733c36_5d16_4f1a_9ef9_0b8f82f2b229.slice/crio-a34c53f06db8c0602e70e230336840a2442f6879fee521a3510e1377f5d9228d WatchSource:0}: Error finding container a34c53f06db8c0602e70e230336840a2442f6879fee521a3510e1377f5d9228d: Status 404 returned error can't find the container with id a34c53f06db8c0602e70e230336840a2442f6879fee521a3510e1377f5d9228d Sep 16 13:22:01.229447 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:01.229415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:01.389453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:01.389419 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" path="/var/lib/kubelet/pods/215fce8d-a522-4be2-b130-fe67a679d22d/volumes" Sep 16 13:22:01.421178 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:01.421132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" event={"ID":"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229","Type":"ContainerStarted","Data":"11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a"} Sep 16 13:22:01.421375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:01.421198 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" event={"ID":"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229","Type":"ContainerStarted","Data":"a34c53f06db8c0602e70e230336840a2442f6879fee521a3510e1377f5d9228d"} Sep 16 13:22:01.421375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:01.421244 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:01.436957 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:01.436906 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" podStartSLOduration=1.4368925049999999 podStartE2EDuration="1.436892505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:01.436434057 +0000 UTC m=+806.556160624" watchObservedRunningTime="2026-09-16 13:22:01.436892505 +0000 UTC m=+806.556619074" Sep 16 13:22:02.025636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:02.025603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:02.722337 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:02.722299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:02.821075 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:02.821044 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:11.512261 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:11.512228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:11.612409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:11.612374 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:11.757246 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:11.757206 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:12.430736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:12.430699 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:17.632312 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:17.632254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:17.737771 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:17.737727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:17.835402 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:17.835368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:18.042363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:18.042323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:21.600958 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:21.600920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:22.339824 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.339773 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:22.750365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.750331 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp"] Sep 16 13:22:22.750774 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.750762 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" Sep 16 13:22:22.750774 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.750775 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" Sep 16 13:22:22.750889 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.750844 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="215fce8d-a522-4be2-b130-fe67a679d22d" containerName="istio-proxy" Sep 16 13:22:22.758870 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.758843 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:22.761243 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.761213 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:22:22.761786 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.761762 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp"] Sep 16 13:22:22.761917 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.761872 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:22:22.762165 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.762146 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:22:22.845668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.845636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c52s8\" (UniqueName: \"kubernetes.io/projected/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-kube-api-access-c52s8\") pod \"maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:22.845668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.845669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:22.946555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.946515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c52s8\" (UniqueName: \"kubernetes.io/projected/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-kube-api-access-c52s8\") pod \"maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:22.946738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.946557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:22.947208 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.947183 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:22.956859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:22.956830 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c52s8\" (UniqueName: \"kubernetes.io/projected/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-kube-api-access-c52s8\") pod \"maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:23.071106 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:23.070984 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:23.225692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:23.225657 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:23.344109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:23.344084 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp"] Sep 16 13:22:23.345692 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:22:23.345661 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e624fe2_2b56_49dc_a8eb_cf6ce32276ba.slice/crio-0e215829a10be866ef8564a22c38cec2442559806aff26af74a23fef7646f672 WatchSource:0}: Error finding container 0e215829a10be866ef8564a22c38cec2442559806aff26af74a23fef7646f672: Status 404 returned error can't find the container with id 0e215829a10be866ef8564a22c38cec2442559806aff26af74a23fef7646f672 Sep 16 13:22:23.552068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:23.552034 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" event={"ID":"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba","Type":"ContainerStarted","Data":"e68d632fd3871a0b5a77ec99d6b00cefccd58e8dd2dfdab0528442dd30572f72"} Sep 16 13:22:23.552235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:23.552086 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" event={"ID":"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba","Type":"ContainerStarted","Data":"0e215829a10be866ef8564a22c38cec2442559806aff26af74a23fef7646f672"} Sep 16 13:22:23.567742 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:23.567698 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" podStartSLOduration=1.567683224 podStartE2EDuration="1.567683224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:23.566987059 +0000 UTC m=+828.686713635" watchObservedRunningTime="2026-09-16 13:22:23.567683224 +0000 UTC m=+828.687409774" Sep 16 13:22:24.116175 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:24.116145 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:24.219892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:24.219773 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:24.559198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:24.559161 2581 generic.go:356] "Generic (PLEG): container finished" podID="0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" containerID="e68d632fd3871a0b5a77ec99d6b00cefccd58e8dd2dfdab0528442dd30572f72" exitCode=0 Sep 16 13:22:24.559415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:24.559246 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" event={"ID":"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba","Type":"ContainerDied","Data":"e68d632fd3871a0b5a77ec99d6b00cefccd58e8dd2dfdab0528442dd30572f72"} Sep 16 13:22:25.070221 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.070176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:25.717166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.717143 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:25.767732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.767709 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c52s8\" (UniqueName: \"kubernetes.io/projected/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-kube-api-access-c52s8\") pod \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " Sep 16 13:22:25.767882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.767747 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-maas-api-service-ca\") pod \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\" (UID: \"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba\") " Sep 16 13:22:25.768114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.768094 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" (UID: "0e624fe2-2b56-49dc-a8eb-cf6ce32276ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:22:25.769734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.769714 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-kube-api-access-c52s8" (OuterVolumeSpecName: "kube-api-access-c52s8") pod "0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" (UID: "0e624fe2-2b56-49dc-a8eb-cf6ce32276ba"). InnerVolumeSpecName "kube-api-access-c52s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:22:25.868923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.868856 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c52s8\" (UniqueName: \"kubernetes.io/projected/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-kube-api-access-c52s8\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:22:25.868923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.868880 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:22:25.923232 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:25.923194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:26.575215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:26.575187 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" Sep 16 13:22:26.575392 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:26.575186 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp" event={"ID":"0e624fe2-2b56-49dc-a8eb-cf6ce32276ba","Type":"ContainerDied","Data":"0e215829a10be866ef8564a22c38cec2442559806aff26af74a23fef7646f672"} Sep 16 13:22:26.575392 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:26.575362 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e215829a10be866ef8564a22c38cec2442559806aff26af74a23fef7646f672" Sep 16 13:22:26.657758 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:26.657727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:27.409614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:27.409580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:28.267078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:28.267046 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:28.920184 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:28.920142 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:28.956140 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:28.956102 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg"] Sep 16 13:22:28.956356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:28.956332 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" podUID="7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" containerName="payload-pre-processing" containerID="cri-o://11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a" gracePeriod=30 Sep 16 13:22:29.019520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.019476 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:29.129644 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.129616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:29.130498 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.130443 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp"] Sep 16 13:22:29.134626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.134600 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-508c5fc8-fhrvp"] Sep 16 13:22:29.200163 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.200072 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:29.219640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.219603 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:29.300680 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.300645 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgcn\" (UniqueName: \"kubernetes.io/projected/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-kube-api-access-5jgcn\") pod \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " Sep 16 13:22:29.300840 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.300745 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-plugins-config-volume\") pod \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\" (UID: \"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229\") " Sep 16 13:22:29.301176 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.301147 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" (UID: "7d733c36-5d16-4f1a-9ef9-0b8f82f2b229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:22:29.302804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.302775 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-kube-api-access-5jgcn" (OuterVolumeSpecName: "kube-api-access-5jgcn") pod "7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" (UID: "7d733c36-5d16-4f1a-9ef9-0b8f82f2b229"). InnerVolumeSpecName "kube-api-access-5jgcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:22:29.303503 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.303481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:29.390123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.390093 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" path="/var/lib/kubelet/pods/0e624fe2-2b56-49dc-a8eb-cf6ce32276ba/volumes" Sep 16 13:22:29.401930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.401906 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:22:29.401930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.401929 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgcn\" (UniqueName: \"kubernetes.io/projected/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229-kube-api-access-5jgcn\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:22:29.596591 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.596556 2581 generic.go:356] "Generic (PLEG): container finished" podID="7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" containerID="11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a" exitCode=0 Sep 16 13:22:29.596799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.596605 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" Sep 16 13:22:29.596799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.596638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" event={"ID":"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229","Type":"ContainerDied","Data":"11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a"} Sep 16 13:22:29.596799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.596689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg" event={"ID":"7d733c36-5d16-4f1a-9ef9-0b8f82f2b229","Type":"ContainerDied","Data":"a34c53f06db8c0602e70e230336840a2442f6879fee521a3510e1377f5d9228d"} Sep 16 13:22:29.596799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.596711 2581 scope.go:117] "RemoveContainer" containerID="11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a" Sep 16 13:22:29.606184 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.606169 2581 scope.go:117] "RemoveContainer" containerID="11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a" Sep 16 13:22:29.606465 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:22:29.606449 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a\": container with ID starting with 11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a not found: ID does not exist" containerID="11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a" Sep 16 13:22:29.606524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.606473 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a"} err="failed to get container status \"11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a\": rpc error: code = NotFound desc = could not find container \"11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a\": container with ID starting with 11f21dea62cc08a08891994a801202528e6f273dcc243e6cccfd0a2a756d1b1a not found: ID does not exist" Sep 16 13:22:29.613646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.613624 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg"] Sep 16 13:22:29.615975 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.615958 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-508c5fc8-589f4667c8-p9rcg"] Sep 16 13:22:29.700600 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.700578 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:29.766492 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.766464 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:29.867045 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:29.866964 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:30.153449 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.153378 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt"] Sep 16 13:22:30.154033 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.154013 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" containerName="revoke-keys" Sep 16 13:22:30.154083 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.154038 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" containerName="revoke-keys" Sep 16 13:22:30.154083 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.154052 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" containerName="payload-pre-processing" Sep 16 13:22:30.154083 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.154060 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" containerName="payload-pre-processing" Sep 16 13:22:30.154179 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.154168 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" containerName="payload-pre-processing" Sep 16 13:22:30.154212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.154185 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e624fe2-2b56-49dc-a8eb-cf6ce32276ba" containerName="revoke-keys" Sep 16 13:22:30.159165 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.159143 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.161973 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.161955 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fa486478-openshift-default-dockercfg-fbwrr\"" Sep 16 13:22:30.168778 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.168755 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt"] Sep 16 13:22:30.189858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.189823 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:30.209112 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-data\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-istio-token\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-envoy\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715c9f-07da-4a15-ab91-36f128409c61-istiod-ca-cert\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-credential-socket\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209339 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715c9f-07da-4a15-ab91-36f128409c61-istio-podinfo\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209427 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-socket\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209427 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209398 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxx6\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-kube-api-access-vgxx6\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.209495 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.209480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-certs\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.262836 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.262802 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:30.310382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715c9f-07da-4a15-ab91-36f128409c61-istiod-ca-cert\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-credential-socket\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715c9f-07da-4a15-ab91-36f128409c61-istio-podinfo\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-socket\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxx6\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-kube-api-access-vgxx6\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-certs\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-data\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-istio-token\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.310817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.310720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-envoy\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.312141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.311437 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-socket\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.312141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.311573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-certs\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.312141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.311812 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-data\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.312141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.312099 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715c9f-07da-4a15-ab91-36f128409c61-istiod-ca-cert\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.312450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.312247 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-credential-socket\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.314953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.314923 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-envoy\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.315338 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.315318 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715c9f-07da-4a15-ab91-36f128409c61-istio-podinfo\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.321235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.321204 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxx6\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-kube-api-access-vgxx6\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.321526 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.321505 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-istio-token\") pod \"e2e-ait-fa486478-openshift-default-55699657d4-85xmt\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.470563 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.470538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:30.752142 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.752067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:30.760564 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.760528 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt"] Sep 16 13:22:30.765599 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:22:30.765575 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2715c9f_07da_4a15_ab91_36f128409c61.slice/crio-b50b358fadfab12294b17ee8d192a251f51eaf24c3952d5086b5cd1882ae9bb1 WatchSource:0}: Error finding container b50b358fadfab12294b17ee8d192a251f51eaf24c3952d5086b5cd1882ae9bb1: Status 404 returned error can't find the container with id b50b358fadfab12294b17ee8d192a251f51eaf24c3952d5086b5cd1882ae9bb1 Sep 16 13:22:30.767831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.767795 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:22:30.767910 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.767861 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:22:30.767910 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:30.767888 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:22:31.025536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:31.025427 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:31.388367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:31.388267 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d733c36-5d16-4f1a-9ef9-0b8f82f2b229" path="/var/lib/kubelet/pods/7d733c36-5d16-4f1a-9ef9-0b8f82f2b229/volumes" Sep 16 13:22:31.611317 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:31.611256 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" event={"ID":"d2715c9f-07da-4a15-ab91-36f128409c61","Type":"ContainerStarted","Data":"fc08fbdc10d3bd77cd7006b7d17f556589299ff3537ddd925e2a89608b09548a"} Sep 16 13:22:31.611504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:31.611330 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" event={"ID":"d2715c9f-07da-4a15-ab91-36f128409c61","Type":"ContainerStarted","Data":"b50b358fadfab12294b17ee8d192a251f51eaf24c3952d5086b5cd1882ae9bb1"} Sep 16 13:22:31.633676 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:31.633632 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podStartSLOduration=1.633618156 podStartE2EDuration="1.633618156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:31.631556441 +0000 UTC m=+836.751283011" watchObservedRunningTime="2026-09-16 13:22:31.633618156 +0000 UTC m=+836.753344705" Sep 16 13:22:31.722138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:31.722084 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:32.470837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:32.470806 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:32.476350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:32.476327 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:32.487151 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:32.487120 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:32.617830 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:32.617793 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:32.618676 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:32.618658 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:33.197523 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:33.197485 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:33.323326 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:33.323268 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:33.406265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:33.406231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:34.163793 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:34.163761 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:35.229517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:35.229481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:36.017978 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:36.017941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:36.716482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:36.716449 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:37.548083 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:37.548044 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:37.648515 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:37.648480 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:37.886437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:37.886346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:38.078059 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.078020 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:38.167857 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.167771 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:22:38.172859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.172829 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf"] Sep 16 13:22:38.178102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.178085 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.183556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.183499 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fa486478-dockercfg-gl22s\"" Sep 16 13:22:38.184277 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.184257 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fa486478\"" Sep 16 13:22:38.192923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.192903 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf"] Sep 16 13:22:38.285248 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.285217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d863c2-c5c1-40d8-8b99-39844e5863d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.285248 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.285257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kfz\" (UniqueName: \"kubernetes.io/projected/67d863c2-c5c1-40d8-8b99-39844e5863d7-kube-api-access-n7kfz\") pod \"payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.386586 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.386553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d863c2-c5c1-40d8-8b99-39844e5863d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.386586 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.386585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kfz\" (UniqueName: \"kubernetes.io/projected/67d863c2-c5c1-40d8-8b99-39844e5863d7-kube-api-access-n7kfz\") pod \"payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.387263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.387239 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d863c2-c5c1-40d8-8b99-39844e5863d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.395043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.395015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kfz\" (UniqueName: \"kubernetes.io/projected/67d863c2-c5c1-40d8-8b99-39844e5863d7-kube-api-access-n7kfz\") pod \"payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.489090 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.489062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:38.768468 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:38.768443 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf"] Sep 16 13:22:38.769378 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:22:38.769354 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d863c2_c5c1_40d8_8b99_39844e5863d7.slice/crio-6e2b94b6435bb23d3cb0044991e63e72cc1e6dfed64b591210819f1524904d37 WatchSource:0}: Error finding container 6e2b94b6435bb23d3cb0044991e63e72cc1e6dfed64b591210819f1524904d37: Status 404 returned error can't find the container with id 6e2b94b6435bb23d3cb0044991e63e72cc1e6dfed64b591210819f1524904d37 Sep 16 13:22:39.661900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:39.661864 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" event={"ID":"67d863c2-c5c1-40d8-8b99-39844e5863d7","Type":"ContainerStarted","Data":"1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771"} Sep 16 13:22:39.662311 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:39.661915 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" event={"ID":"67d863c2-c5c1-40d8-8b99-39844e5863d7","Type":"ContainerStarted","Data":"6e2b94b6435bb23d3cb0044991e63e72cc1e6dfed64b591210819f1524904d37"} Sep 16 13:22:39.662311 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:39.661959 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:39.680530 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:39.680488 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" podStartSLOduration=1.680477241 podStartE2EDuration="1.680477241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:39.677232481 +0000 UTC m=+844.796959041" watchObservedRunningTime="2026-09-16 13:22:39.680477241 +0000 UTC m=+844.800203789" Sep 16 13:22:50.669195 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:50.669163 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:22:52.748273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:52.748241 2581 generic.go:356] "Generic (PLEG): container finished" podID="d2715c9f-07da-4a15-ab91-36f128409c61" containerID="fc08fbdc10d3bd77cd7006b7d17f556589299ff3537ddd925e2a89608b09548a" exitCode=0 Sep 16 13:22:52.748668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:52.748315 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" event={"ID":"d2715c9f-07da-4a15-ab91-36f128409c61","Type":"ContainerDied","Data":"fc08fbdc10d3bd77cd7006b7d17f556589299ff3537ddd925e2a89608b09548a"} Sep 16 13:22:52.748765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:52.748746 2581 scope.go:117] "RemoveContainer" containerID="fc08fbdc10d3bd77cd7006b7d17f556589299ff3537ddd925e2a89608b09548a" Sep 16 13:22:52.749915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:52.749882 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:22:52.749980 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:52.749962 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:22:52.750017 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:52.750010 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:22:53.761004 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:53.760969 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" event={"ID":"d2715c9f-07da-4a15-ab91-36f128409c61","Type":"ContainerStarted","Data":"51d3528589fd6ef600b423e7d0918e7b3dd29ba2401e23b9c2efdd4b4760b494"} Sep 16 13:22:54.471114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:54.471082 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:54.471328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:54.471270 2581 patch_prober.go:28] interesting pod/e2e-ait-fa486478-openshift-default-55699657d4-85xmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 16 13:22:54.471386 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:54.471336 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 16 13:22:55.471275 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:55.471240 2581 patch_prober.go:28] interesting pod/e2e-ait-fa486478-openshift-default-55699657d4-85xmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 16 13:22:55.471702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:55.471319 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 16 13:22:56.471732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:56.471698 2581 patch_prober.go:28] interesting pod/e2e-ait-fa486478-openshift-default-55699657d4-85xmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 16 13:22:56.472192 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:56.471767 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 16 13:22:57.471562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:57.471517 2581 patch_prober.go:28] interesting pod/e2e-ait-fa486478-openshift-default-55699657d4-85xmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 16 13:22:57.471771 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:57.471581 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 16 13:22:58.470858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:58.470824 2581 patch_prober.go:28] interesting pod/e2e-ait-fa486478-openshift-default-55699657d4-85xmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 16 13:22:58.471037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:58.470877 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 16 13:22:59.474795 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:59.474767 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:59.475211 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:59.475056 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:59.475835 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:59.475818 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:22:59.636985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:22:59.636949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:00.439486 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:00.439442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:01.129752 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:01.129720 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:01.931949 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:01.931914 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:02.624237 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:02.624202 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:02.785387 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:02.785352 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:02.886119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:02.886031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:03.317915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:03.317883 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:04.147696 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.147661 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:04.401243 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.401149 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm"] Sep 16 13:23:04.408551 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.408528 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.410900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.410874 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:23:04.411149 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.411112 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:23:04.411723 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.411697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm"] Sep 16 13:23:04.412037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.411971 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:23:04.529641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.529610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066ffe74-a3c8-4f20-bec9-f890a0d43342-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fa486478-sz5dm\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.529641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.529642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786vf\" (UniqueName: \"kubernetes.io/projected/066ffe74-a3c8-4f20-bec9-f890a0d43342-kube-api-access-786vf\") pod \"maas-api-revoke-keys-e2e-ait-fa486478-sz5dm\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.630194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.630148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066ffe74-a3c8-4f20-bec9-f890a0d43342-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fa486478-sz5dm\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.630194 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.630192 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786vf\" (UniqueName: \"kubernetes.io/projected/066ffe74-a3c8-4f20-bec9-f890a0d43342-kube-api-access-786vf\") pod \"maas-api-revoke-keys-e2e-ait-fa486478-sz5dm\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.630768 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.630749 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066ffe74-a3c8-4f20-bec9-f890a0d43342-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fa486478-sz5dm\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.638404 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.638375 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-786vf\" (UniqueName: \"kubernetes.io/projected/066ffe74-a3c8-4f20-bec9-f890a0d43342-kube-api-access-786vf\") pod \"maas-api-revoke-keys-e2e-ait-fa486478-sz5dm\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.721565 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.721542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:04.946070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.946031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:04.994034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:04.994011 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm"] Sep 16 13:23:04.996295 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:23:04.996247 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066ffe74_a3c8_4f20_bec9_f890a0d43342.slice/crio-a4872c9698b78dad4cebff60cb951d05fe6275a0d04ba7f08d17bb6cbca9a4c5 WatchSource:0}: Error finding container a4872c9698b78dad4cebff60cb951d05fe6275a0d04ba7f08d17bb6cbca9a4c5: Status 404 returned error can't find the container with id a4872c9698b78dad4cebff60cb951d05fe6275a0d04ba7f08d17bb6cbca9a4c5 Sep 16 13:23:05.722182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:05.722142 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:05.833936 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:05.833908 2581 generic.go:356] "Generic (PLEG): container finished" podID="066ffe74-a3c8-4f20-bec9-f890a0d43342" containerID="d82af04e72837affbd851d67bb73aa260ca4279db79fd4be7e86f79fc197c852" exitCode=0 Sep 16 13:23:05.834058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:05.834040 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" event={"ID":"066ffe74-a3c8-4f20-bec9-f890a0d43342","Type":"ContainerDied","Data":"d82af04e72837affbd851d67bb73aa260ca4279db79fd4be7e86f79fc197c852"} Sep 16 13:23:05.834104 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:05.834088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" event={"ID":"066ffe74-a3c8-4f20-bec9-f890a0d43342","Type":"ContainerStarted","Data":"a4872c9698b78dad4cebff60cb951d05fe6275a0d04ba7f08d17bb6cbca9a4c5"} Sep 16 13:23:06.533200 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:06.533171 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:06.979682 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:06.979652 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:07.151308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.151200 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066ffe74-a3c8-4f20-bec9-f890a0d43342-maas-api-service-ca\") pod \"066ffe74-a3c8-4f20-bec9-f890a0d43342\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " Sep 16 13:23:07.151466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.151358 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-786vf\" (UniqueName: \"kubernetes.io/projected/066ffe74-a3c8-4f20-bec9-f890a0d43342-kube-api-access-786vf\") pod \"066ffe74-a3c8-4f20-bec9-f890a0d43342\" (UID: \"066ffe74-a3c8-4f20-bec9-f890a0d43342\") " Sep 16 13:23:07.151579 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.151557 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066ffe74-a3c8-4f20-bec9-f890a0d43342-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "066ffe74-a3c8-4f20-bec9-f890a0d43342" (UID: "066ffe74-a3c8-4f20-bec9-f890a0d43342"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:07.153497 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.153478 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066ffe74-a3c8-4f20-bec9-f890a0d43342-kube-api-access-786vf" (OuterVolumeSpecName: "kube-api-access-786vf") pod "066ffe74-a3c8-4f20-bec9-f890a0d43342" (UID: "066ffe74-a3c8-4f20-bec9-f890a0d43342"). InnerVolumeSpecName "kube-api-access-786vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:07.218689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.218651 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:07.252197 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.252167 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786vf\" (UniqueName: \"kubernetes.io/projected/066ffe74-a3c8-4f20-bec9-f890a0d43342-kube-api-access-786vf\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:07.252197 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.252194 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066ffe74-a3c8-4f20-bec9-f890a0d43342-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:07.846174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.846144 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" Sep 16 13:23:07.846378 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.846185 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm" event={"ID":"066ffe74-a3c8-4f20-bec9-f890a0d43342","Type":"ContainerDied","Data":"a4872c9698b78dad4cebff60cb951d05fe6275a0d04ba7f08d17bb6cbca9a4c5"} Sep 16 13:23:07.846378 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:07.846230 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4872c9698b78dad4cebff60cb951d05fe6275a0d04ba7f08d17bb6cbca9a4c5" Sep 16 13:23:08.021307 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:08.021264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:08.830150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:08.830107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:09.498437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:09.498403 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:09.602730 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:09.602698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.229561 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.229525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.278488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.278457 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf"] Sep 16 13:23:15.278677 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.278654 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" podUID="67d863c2-c5c1-40d8-8b99-39844e5863d7" containerName="payload-pre-processing" containerID="cri-o://1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771" gracePeriod=30 Sep 16 13:23:15.319438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.319404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.399441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.399393 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.488768 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.488683 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm"] Sep 16 13:23:15.494710 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.494683 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fa486478-sz5dm"] Sep 16 13:23:15.495409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.495380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.538966 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.538946 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:23:15.577075 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.576994 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.633907 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.633877 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d863c2-c5c1-40d8-8b99-39844e5863d7-plugins-config-volume\") pod \"67d863c2-c5c1-40d8-8b99-39844e5863d7\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " Sep 16 13:23:15.634079 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.633916 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7kfz\" (UniqueName: \"kubernetes.io/projected/67d863c2-c5c1-40d8-8b99-39844e5863d7-kube-api-access-n7kfz\") pod \"67d863c2-c5c1-40d8-8b99-39844e5863d7\" (UID: \"67d863c2-c5c1-40d8-8b99-39844e5863d7\") " Sep 16 13:23:15.634373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.634351 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d863c2-c5c1-40d8-8b99-39844e5863d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67d863c2-c5c1-40d8-8b99-39844e5863d7" (UID: "67d863c2-c5c1-40d8-8b99-39844e5863d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:15.636018 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.635999 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d863c2-c5c1-40d8-8b99-39844e5863d7-kube-api-access-n7kfz" (OuterVolumeSpecName: "kube-api-access-n7kfz") pod "67d863c2-c5c1-40d8-8b99-39844e5863d7" (UID: "67d863c2-c5c1-40d8-8b99-39844e5863d7"). InnerVolumeSpecName "kube-api-access-n7kfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:15.735020 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.734983 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d863c2-c5c1-40d8-8b99-39844e5863d7-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:15.735020 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.735013 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7kfz\" (UniqueName: \"kubernetes.io/projected/67d863c2-c5c1-40d8-8b99-39844e5863d7-kube-api-access-n7kfz\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:15.897315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.897211 2581 generic.go:356] "Generic (PLEG): container finished" podID="67d863c2-c5c1-40d8-8b99-39844e5863d7" containerID="1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771" exitCode=0 Sep 16 13:23:15.897315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.897254 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" Sep 16 13:23:15.897557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.897318 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" event={"ID":"67d863c2-c5c1-40d8-8b99-39844e5863d7","Type":"ContainerDied","Data":"1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771"} Sep 16 13:23:15.897557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.897379 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf" event={"ID":"67d863c2-c5c1-40d8-8b99-39844e5863d7","Type":"ContainerDied","Data":"6e2b94b6435bb23d3cb0044991e63e72cc1e6dfed64b591210819f1524904d37"} Sep 16 13:23:15.897557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.897408 2581 scope.go:117] "RemoveContainer" containerID="1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771" Sep 16 13:23:15.906555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.906531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:15.906912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.906897 2581 scope.go:117] "RemoveContainer" containerID="1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771" Sep 16 13:23:15.907173 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:23:15.907156 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771\": container with ID starting with 1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771 not found: ID does not exist" containerID="1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771" Sep 16 13:23:15.907243 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.907180 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771"} err="failed to get container status \"1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771\": rpc error: code = NotFound desc = could not find container \"1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771\": container with ID starting with 1ba935477c857b7c8e2bb508990831902695c18a19ed785a14889aea7f018771 not found: ID does not exist" Sep 16 13:23:15.921611 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.921587 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf"] Sep 16 13:23:15.924044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:15.924026 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fa486478-554cf74ccd-c77bf"] Sep 16 13:23:16.696489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:16.696452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:16.795920 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:16.795885 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt"] Sep 16 13:23:16.796102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:16.796078 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" containerID="cri-o://51d3528589fd6ef600b423e7d0918e7b3dd29ba2401e23b9c2efdd4b4760b494" gracePeriod=30 Sep 16 13:23:16.803720 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:16.803692 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:17.292552 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:17.292513 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:17.388394 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:17.388357 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066ffe74-a3c8-4f20-bec9-f890a0d43342" path="/var/lib/kubelet/pods/066ffe74-a3c8-4f20-bec9-f890a0d43342/volumes" Sep 16 13:23:17.388825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:17.388806 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d863c2-c5c1-40d8-8b99-39844e5863d7" path="/var/lib/kubelet/pods/67d863c2-c5c1-40d8-8b99-39844e5863d7/volumes" Sep 16 13:23:17.423846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:17.423814 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:17.636771 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:17.636687 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:18.170610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:18.170580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:18.467308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:18.467251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:19.398686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:19.398648 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:20.188493 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:20.188465 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:20.281853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:20.281821 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:21.249079 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:21.249034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:21.940732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:21.940699 2581 generic.go:356] "Generic (PLEG): container finished" podID="d2715c9f-07da-4a15-ab91-36f128409c61" containerID="51d3528589fd6ef600b423e7d0918e7b3dd29ba2401e23b9c2efdd4b4760b494" exitCode=0 Sep 16 13:23:21.940910 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:21.940768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" event={"ID":"d2715c9f-07da-4a15-ab91-36f128409c61","Type":"ContainerDied","Data":"51d3528589fd6ef600b423e7d0918e7b3dd29ba2401e23b9c2efdd4b4760b494"} Sep 16 13:23:21.940910 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:21.940832 2581 scope.go:117] "RemoveContainer" containerID="fc08fbdc10d3bd77cd7006b7d17f556589299ff3537ddd925e2a89608b09548a" Sep 16 13:23:22.052055 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.052034 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:23:22.095415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095331 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-istio-token\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095414 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-credential-socket\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095459 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-certs\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095484 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-data\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095531 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-socket\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095556 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgxx6\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-kube-api-access-vgxx6\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095591 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715c9f-07da-4a15-ab91-36f128409c61-istio-podinfo\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-envoy\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095708 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715c9f-07da-4a15-ab91-36f128409c61-istiod-ca-cert\") pod \"d2715c9f-07da-4a15-ab91-36f128409c61\" (UID: \"d2715c9f-07da-4a15-ab91-36f128409c61\") " Sep 16 13:23:22.095900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095784 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:23:22.095900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.095801 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:23:22.096166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.096042 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.096166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.096060 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.096300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.096192 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:23:22.096300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.096262 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:23:22.096430 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.096152 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2715c9f-07da-4a15-ab91-36f128409c61-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:22.097848 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.097820 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:22.098104 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.098086 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2715c9f-07da-4a15-ab91-36f128409c61-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:23:22.098268 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.098252 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:23:22.098565 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.098539 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-kube-api-access-vgxx6" (OuterVolumeSpecName: "kube-api-access-vgxx6") pod "d2715c9f-07da-4a15-ab91-36f128409c61" (UID: "d2715c9f-07da-4a15-ab91-36f128409c61"). InnerVolumeSpecName "kube-api-access-vgxx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:22.141390 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.141357 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:22.196695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196666 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715c9f-07da-4a15-ab91-36f128409c61-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.196695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196692 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.196695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196700 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.196911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196709 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.196911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196719 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxx6\" (UniqueName: \"kubernetes.io/projected/d2715c9f-07da-4a15-ab91-36f128409c61-kube-api-access-vgxx6\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.196911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196727 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715c9f-07da-4a15-ab91-36f128409c61-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.196911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.196735 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715c9f-07da-4a15-ab91-36f128409c61-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:22.948264 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.948233 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" Sep 16 13:23:22.948709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.948274 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt" event={"ID":"d2715c9f-07da-4a15-ab91-36f128409c61","Type":"ContainerDied","Data":"b50b358fadfab12294b17ee8d192a251f51eaf24c3952d5086b5cd1882ae9bb1"} Sep 16 13:23:22.948709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.948356 2581 scope.go:117] "RemoveContainer" containerID="51d3528589fd6ef600b423e7d0918e7b3dd29ba2401e23b9c2efdd4b4760b494" Sep 16 13:23:22.970448 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.970415 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt"] Sep 16 13:23:22.975260 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.975239 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fa486478-openshift-default-55699657d4-85xmt"] Sep 16 13:23:22.989926 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:22.989889 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:23.389185 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:23.389148 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" path="/var/lib/kubelet/pods/d2715c9f-07da-4a15-ab91-36f128409c61/volumes" Sep 16 13:23:23.870707 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:23.870666 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:24.558350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:24.558315 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:25.351196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.351165 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:25.458429 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.458400 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:25.550779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.550735 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:25.862613 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.862585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:25.970121 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.970082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:25.990081 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990052 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt"] Sep 16 13:23:25.990485 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990472 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" Sep 16 13:23:25.990535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990487 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" Sep 16 13:23:25.990535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990505 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d863c2-c5c1-40d8-8b99-39844e5863d7" containerName="payload-pre-processing" Sep 16 13:23:25.990535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990510 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d863c2-c5c1-40d8-8b99-39844e5863d7" containerName="payload-pre-processing" Sep 16 13:23:25.990535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990520 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" Sep 16 13:23:25.990535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990526 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" Sep 16 13:23:25.990535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990532 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066ffe74-a3c8-4f20-bec9-f890a0d43342" containerName="revoke-keys" Sep 16 13:23:25.990716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990537 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066ffe74-a3c8-4f20-bec9-f890a0d43342" containerName="revoke-keys" Sep 16 13:23:25.990716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990602 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" Sep 16 13:23:25.990716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990610 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d863c2-c5c1-40d8-8b99-39844e5863d7" containerName="payload-pre-processing" Sep 16 13:23:25.990716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990616 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2715c9f-07da-4a15-ab91-36f128409c61" containerName="istio-proxy" Sep 16 13:23:25.990716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.990622 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="066ffe74-a3c8-4f20-bec9-f890a0d43342" containerName="revoke-keys" Sep 16 13:23:25.995098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.995071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:25.997992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:25.997957 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9615f1d6\"" Sep 16 13:23:26.005562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.005538 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt"] Sep 16 13:23:26.132718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.132634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fxl\" (UniqueName: \"kubernetes.io/projected/4b236bd1-4b0f-46ff-9640-c93e06caee20-kube-api-access-d6fxl\") pod \"payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.132890 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.132785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b236bd1-4b0f-46ff-9640-c93e06caee20-plugins-config-volume\") pod \"payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.233643 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.233598 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fxl\" (UniqueName: \"kubernetes.io/projected/4b236bd1-4b0f-46ff-9640-c93e06caee20-kube-api-access-d6fxl\") pod \"payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.233832 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.233697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b236bd1-4b0f-46ff-9640-c93e06caee20-plugins-config-volume\") pod \"payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.234464 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.234445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b236bd1-4b0f-46ff-9640-c93e06caee20-plugins-config-volume\") pod \"payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.242731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.242712 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fxl\" (UniqueName: \"kubernetes.io/projected/4b236bd1-4b0f-46ff-9640-c93e06caee20-kube-api-access-d6fxl\") pod \"payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.307497 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.307465 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.580045 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.580023 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt"] Sep 16 13:23:26.581376 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:23:26.581345 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b236bd1_4b0f_46ff_9640_c93e06caee20.slice/crio-87cbb6404fcd42423232cf3422251bfe8204bb25abdd8ff1d403f0b374baee97 WatchSource:0}: Error finding container 87cbb6404fcd42423232cf3422251bfe8204bb25abdd8ff1d403f0b374baee97: Status 404 returned error can't find the container with id 87cbb6404fcd42423232cf3422251bfe8204bb25abdd8ff1d403f0b374baee97 Sep 16 13:23:26.976227 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.976185 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" event={"ID":"4b236bd1-4b0f-46ff-9640-c93e06caee20","Type":"ContainerStarted","Data":"b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617"} Sep 16 13:23:26.976635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.976246 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" event={"ID":"4b236bd1-4b0f-46ff-9640-c93e06caee20","Type":"ContainerStarted","Data":"87cbb6404fcd42423232cf3422251bfe8204bb25abdd8ff1d403f0b374baee97"} Sep 16 13:23:26.976635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.976425 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:26.992830 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:26.992773 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" podStartSLOduration=1.992752356 podStartE2EDuration="1.992752356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:26.991519948 +0000 UTC m=+892.111246500" watchObservedRunningTime="2026-09-16 13:23:26.992752356 +0000 UTC m=+892.112478909" Sep 16 13:23:28.907638 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:28.907563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:30.066306 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:30.066247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:30.133203 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:30.133165 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:31.686353 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:31.686260 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:35.393448 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:35.393422 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:23:35.395041 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:35.395020 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:23:37.984622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:37.984586 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:23:45.233431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.233399 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68"] Sep 16 13:23:45.238358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.238341 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.240667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.240645 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:23:45.240796 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.240740 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:23:45.241478 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.241460 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:23:45.247771 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.247745 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68"] Sep 16 13:23:45.399731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.399693 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:45.404734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.404712 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b7e5d-fccc-4a10-975e-e84bbc455796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-15da7b-pfn68\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.404853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.404781 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7dr\" (UniqueName: \"kubernetes.io/projected/309b7e5d-fccc-4a10-975e-e84bbc455796-kube-api-access-2p7dr\") pod \"maas-api-revoke-keys-e2e-autores-15da7b-pfn68\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.505302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.505186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7dr\" (UniqueName: \"kubernetes.io/projected/309b7e5d-fccc-4a10-975e-e84bbc455796-kube-api-access-2p7dr\") pod \"maas-api-revoke-keys-e2e-autores-15da7b-pfn68\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.505302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.505270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b7e5d-fccc-4a10-975e-e84bbc455796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-15da7b-pfn68\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.505882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.505863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b7e5d-fccc-4a10-975e-e84bbc455796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-15da7b-pfn68\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.513317 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.513292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7dr\" (UniqueName: \"kubernetes.io/projected/309b7e5d-fccc-4a10-975e-e84bbc455796-kube-api-access-2p7dr\") pod \"maas-api-revoke-keys-e2e-autores-15da7b-pfn68\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.551358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.551339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:45.833501 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:45.833470 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68"] Sep 16 13:23:45.834736 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:23:45.834713 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309b7e5d_fccc_4a10_975e_e84bbc455796.slice/crio-f494d9ae28f9eaceed5891c34ce28d7e320ea017570f847daf7f41eded8f2dfc WatchSource:0}: Error finding container f494d9ae28f9eaceed5891c34ce28d7e320ea017570f847daf7f41eded8f2dfc: Status 404 returned error can't find the container with id f494d9ae28f9eaceed5891c34ce28d7e320ea017570f847daf7f41eded8f2dfc Sep 16 13:23:46.090601 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:46.090568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" event={"ID":"309b7e5d-fccc-4a10-975e-e84bbc455796","Type":"ContainerStarted","Data":"f494d9ae28f9eaceed5891c34ce28d7e320ea017570f847daf7f41eded8f2dfc"} Sep 16 13:23:46.320882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:46.320836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:47.098554 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:47.098522 2581 generic.go:356] "Generic (PLEG): container finished" podID="309b7e5d-fccc-4a10-975e-e84bbc455796" containerID="d862b5680798cd5d06b7e05501844f8c9a4c2cac3e433e4f4e384686248591a2" exitCode=0 Sep 16 13:23:47.098743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:47.098608 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" event={"ID":"309b7e5d-fccc-4a10-975e-e84bbc455796","Type":"ContainerDied","Data":"d862b5680798cd5d06b7e05501844f8c9a4c2cac3e433e4f4e384686248591a2"} Sep 16 13:23:47.263403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:47.263373 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:48.241381 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.241358 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:48.278719 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.278677 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:48.332503 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.332463 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b7e5d-fccc-4a10-975e-e84bbc455796-maas-api-service-ca\") pod \"309b7e5d-fccc-4a10-975e-e84bbc455796\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " Sep 16 13:23:48.332861 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.332823 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p7dr\" (UniqueName: \"kubernetes.io/projected/309b7e5d-fccc-4a10-975e-e84bbc455796-kube-api-access-2p7dr\") pod \"309b7e5d-fccc-4a10-975e-e84bbc455796\" (UID: \"309b7e5d-fccc-4a10-975e-e84bbc455796\") " Sep 16 13:23:48.332987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.332887 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309b7e5d-fccc-4a10-975e-e84bbc455796-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "309b7e5d-fccc-4a10-975e-e84bbc455796" (UID: "309b7e5d-fccc-4a10-975e-e84bbc455796"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:48.333150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.333126 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/309b7e5d-fccc-4a10-975e-e84bbc455796-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:48.335276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.335254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309b7e5d-fccc-4a10-975e-e84bbc455796-kube-api-access-2p7dr" (OuterVolumeSpecName: "kube-api-access-2p7dr") pod "309b7e5d-fccc-4a10-975e-e84bbc455796" (UID: "309b7e5d-fccc-4a10-975e-e84bbc455796"). InnerVolumeSpecName "kube-api-access-2p7dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:48.434126 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:48.434042 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7dr\" (UniqueName: \"kubernetes.io/projected/309b7e5d-fccc-4a10-975e-e84bbc455796-kube-api-access-2p7dr\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:49.112074 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:49.112049 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" Sep 16 13:23:49.112259 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:49.112039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68" event={"ID":"309b7e5d-fccc-4a10-975e-e84bbc455796","Type":"ContainerDied","Data":"f494d9ae28f9eaceed5891c34ce28d7e320ea017570f847daf7f41eded8f2dfc"} Sep 16 13:23:49.112259 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:49.112176 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f494d9ae28f9eaceed5891c34ce28d7e320ea017570f847daf7f41eded8f2dfc" Sep 16 13:23:49.249425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:49.249394 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:50.213042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.213009 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg"] Sep 16 13:23:50.213692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.213665 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309b7e5d-fccc-4a10-975e-e84bbc455796" containerName="revoke-keys" Sep 16 13:23:50.213692 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.213687 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="309b7e5d-fccc-4a10-975e-e84bbc455796" containerName="revoke-keys" Sep 16 13:23:50.213908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.213804 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="309b7e5d-fccc-4a10-975e-e84bbc455796" containerName="revoke-keys" Sep 16 13:23:50.217626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.217606 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.222431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.222410 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:23:50.222543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.222428 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:23:50.223067 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.223053 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:23:50.230457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.230434 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg"] Sep 16 13:23:50.354328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.354273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdr6\" (UniqueName: \"kubernetes.io/projected/42f28702-ad85-4002-8491-7442e22e344d-kube-api-access-sxdr6\") pod \"maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.354718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.354381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f28702-ad85-4002-8491-7442e22e344d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.455625 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.455592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdr6\" (UniqueName: \"kubernetes.io/projected/42f28702-ad85-4002-8491-7442e22e344d-kube-api-access-sxdr6\") pod \"maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.455799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.455642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f28702-ad85-4002-8491-7442e22e344d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.456314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.456260 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f28702-ad85-4002-8491-7442e22e344d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.464396 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.464339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdr6\" (UniqueName: \"kubernetes.io/projected/42f28702-ad85-4002-8491-7442e22e344d-kube-api-access-sxdr6\") pod \"maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.529482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.529457 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:50.646158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.646127 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:50.800968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.800943 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg"] Sep 16 13:23:50.802423 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:23:50.802396 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f28702_ad85_4002_8491_7442e22e344d.slice/crio-0599c6e4d263f898b72c4f5aec5b64c1ea14b573cf3815901f837ccf1a0653ed WatchSource:0}: Error finding container 0599c6e4d263f898b72c4f5aec5b64c1ea14b573cf3815901f837ccf1a0653ed: Status 404 returned error can't find the container with id 0599c6e4d263f898b72c4f5aec5b64c1ea14b573cf3815901f837ccf1a0653ed Sep 16 13:23:50.978718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:50.978682 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:51.126636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:51.126603 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" event={"ID":"42f28702-ad85-4002-8491-7442e22e344d","Type":"ContainerStarted","Data":"0599c6e4d263f898b72c4f5aec5b64c1ea14b573cf3815901f837ccf1a0653ed"} Sep 16 13:23:51.824110 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:51.824086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:52.134616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:52.134582 2581 generic.go:356] "Generic (PLEG): container finished" podID="42f28702-ad85-4002-8491-7442e22e344d" containerID="cf56b139918243a215ce565ce864f61a71c1ef22ef5ded512d55bdfdbcbe897d" exitCode=0 Sep 16 13:23:52.134789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:52.134630 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" event={"ID":"42f28702-ad85-4002-8491-7442e22e344d","Type":"ContainerDied","Data":"cf56b139918243a215ce565ce864f61a71c1ef22ef5ded512d55bdfdbcbe897d"} Sep 16 13:23:52.558154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:52.558120 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:53.276200 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.276175 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:53.282581 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.282557 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxdr6\" (UniqueName: \"kubernetes.io/projected/42f28702-ad85-4002-8491-7442e22e344d-kube-api-access-sxdr6\") pod \"42f28702-ad85-4002-8491-7442e22e344d\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " Sep 16 13:23:53.282652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.282595 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f28702-ad85-4002-8491-7442e22e344d-maas-api-service-ca\") pod \"42f28702-ad85-4002-8491-7442e22e344d\" (UID: \"42f28702-ad85-4002-8491-7442e22e344d\") " Sep 16 13:23:53.283019 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.282993 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f28702-ad85-4002-8491-7442e22e344d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42f28702-ad85-4002-8491-7442e22e344d" (UID: "42f28702-ad85-4002-8491-7442e22e344d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:53.284683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.284661 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f28702-ad85-4002-8491-7442e22e344d-kube-api-access-sxdr6" (OuterVolumeSpecName: "kube-api-access-sxdr6") pod "42f28702-ad85-4002-8491-7442e22e344d" (UID: "42f28702-ad85-4002-8491-7442e22e344d"). InnerVolumeSpecName "kube-api-access-sxdr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:53.298603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.298575 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:53.383551 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.383515 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdr6\" (UniqueName: \"kubernetes.io/projected/42f28702-ad85-4002-8491-7442e22e344d-kube-api-access-sxdr6\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:53.383551 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:53.383552 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f28702-ad85-4002-8491-7442e22e344d-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:54.149137 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:54.149104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" event={"ID":"42f28702-ad85-4002-8491-7442e22e344d","Type":"ContainerDied","Data":"0599c6e4d263f898b72c4f5aec5b64c1ea14b573cf3815901f837ccf1a0653ed"} Sep 16 13:23:54.149137 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:54.149129 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg" Sep 16 13:23:54.149382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:54.149157 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0599c6e4d263f898b72c4f5aec5b64c1ea14b573cf3815901f837ccf1a0653ed" Sep 16 13:23:54.234748 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:54.234710 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:55.033563 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:55.033522 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:55.912978 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:55.912942 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:56.520183 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.520148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:56.555203 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.555173 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng"] Sep 16 13:23:56.555517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.555489 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" podUID="fccbedcb-9793-4447-a09b-89ff6c1e6179" containerName="payload-processing" containerID="cri-o://177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5" gracePeriod=30 Sep 16 13:23:56.570595 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.570568 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq"] Sep 16 13:23:56.570817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.570793 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" podUID="4cbcbd82-0d67-4a36-b089-964b63c3a2d2" containerName="payload-pre-processing" containerID="cri-o://b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f" gracePeriod=30 Sep 16 13:23:56.621222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.621030 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:56.717807 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.717774 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:56.762401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.762277 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68"] Sep 16 13:23:56.770842 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.770773 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-15da7b-pfn68"] Sep 16 13:23:56.792952 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.792918 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:56.848347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.848323 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:23:56.851304 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.851270 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:23:56.910482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.910452 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9848m\" (UniqueName: \"kubernetes.io/projected/fccbedcb-9793-4447-a09b-89ff6c1e6179-kube-api-access-9848m\") pod \"fccbedcb-9793-4447-a09b-89ff6c1e6179\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " Sep 16 13:23:56.910672 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.910505 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-plugins-config-volume\") pod \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " Sep 16 13:23:56.911155 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.911124 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cbcbd82-0d67-4a36-b089-964b63c3a2d2" (UID: "4cbcbd82-0d67-4a36-b089-964b63c3a2d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:56.912765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:56.912741 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccbedcb-9793-4447-a09b-89ff6c1e6179-kube-api-access-9848m" (OuterVolumeSpecName: "kube-api-access-9848m") pod "fccbedcb-9793-4447-a09b-89ff6c1e6179" (UID: "fccbedcb-9793-4447-a09b-89ff6c1e6179"). InnerVolumeSpecName "kube-api-access-9848m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:57.011096 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.011061 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjmnb\" (UniqueName: \"kubernetes.io/projected/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-kube-api-access-pjmnb\") pod \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\" (UID: \"4cbcbd82-0d67-4a36-b089-964b63c3a2d2\") " Sep 16 13:23:57.011096 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.011100 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccbedcb-9793-4447-a09b-89ff6c1e6179-plugins-config-volume\") pod \"fccbedcb-9793-4447-a09b-89ff6c1e6179\" (UID: \"fccbedcb-9793-4447-a09b-89ff6c1e6179\") " Sep 16 13:23:57.011485 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.011467 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9848m\" (UniqueName: \"kubernetes.io/projected/fccbedcb-9793-4447-a09b-89ff6c1e6179-kube-api-access-9848m\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:57.011528 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.011477 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccbedcb-9793-4447-a09b-89ff6c1e6179-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fccbedcb-9793-4447-a09b-89ff6c1e6179" (UID: "fccbedcb-9793-4447-a09b-89ff6c1e6179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:23:57.011528 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.011492 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:57.013148 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.013131 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-kube-api-access-pjmnb" (OuterVolumeSpecName: "kube-api-access-pjmnb") pod "4cbcbd82-0d67-4a36-b089-964b63c3a2d2" (UID: "4cbcbd82-0d67-4a36-b089-964b63c3a2d2"). InnerVolumeSpecName "kube-api-access-pjmnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:23:57.112103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.112036 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjmnb\" (UniqueName: \"kubernetes.io/projected/4cbcbd82-0d67-4a36-b089-964b63c3a2d2-kube-api-access-pjmnb\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:57.112103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.112058 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccbedcb-9793-4447-a09b-89ff6c1e6179-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:23:57.168931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.168901 2581 generic.go:356] "Generic (PLEG): container finished" podID="4cbcbd82-0d67-4a36-b089-964b63c3a2d2" containerID="b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f" exitCode=0 Sep 16 13:23:57.169105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.168940 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" Sep 16 13:23:57.169105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.168988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" event={"ID":"4cbcbd82-0d67-4a36-b089-964b63c3a2d2","Type":"ContainerDied","Data":"b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f"} Sep 16 13:23:57.169105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.169044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq" event={"ID":"4cbcbd82-0d67-4a36-b089-964b63c3a2d2","Type":"ContainerDied","Data":"96f68b9ae74c2725738350c2082145df427c278a285b30b70f49f6b7d2fc1e51"} Sep 16 13:23:57.169105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.169073 2581 scope.go:117] "RemoveContainer" containerID="b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f" Sep 16 13:23:57.170325 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.170272 2581 generic.go:356] "Generic (PLEG): container finished" podID="fccbedcb-9793-4447-a09b-89ff6c1e6179" containerID="177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5" exitCode=0 Sep 16 13:23:57.170413 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.170326 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" Sep 16 13:23:57.170413 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.170331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" event={"ID":"fccbedcb-9793-4447-a09b-89ff6c1e6179","Type":"ContainerDied","Data":"177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5"} Sep 16 13:23:57.170413 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.170377 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng" event={"ID":"fccbedcb-9793-4447-a09b-89ff6c1e6179","Type":"ContainerDied","Data":"822d986406a2c674432f27cf64be7d883d3397e82b19922d827388e2b4d64e58"} Sep 16 13:23:57.178937 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.178915 2581 scope.go:117] "RemoveContainer" containerID="b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f" Sep 16 13:23:57.179248 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:23:57.179229 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f\": container with ID starting with b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f not found: ID does not exist" containerID="b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f" Sep 16 13:23:57.179362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.179260 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f"} err="failed to get container status \"b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f\": rpc error: code = NotFound desc = could not find container \"b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f\": container with ID starting with b190e7a9b3647ff50cad3b853a5ba7c8cbae5ae3821136ef37a74e4b8325455f not found: ID does not exist" Sep 16 13:23:57.179362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.179298 2581 scope.go:117] "RemoveContainer" containerID="177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5" Sep 16 13:23:57.187961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.187941 2581 scope.go:117] "RemoveContainer" containerID="177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5" Sep 16 13:23:57.188252 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:23:57.188233 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5\": container with ID starting with 177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5 not found: ID does not exist" containerID="177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5" Sep 16 13:23:57.188347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.188258 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5"} err="failed to get container status \"177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5\": rpc error: code = NotFound desc = could not find container \"177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5\": container with ID starting with 177b7ff52e2670096bc01c3b30b76b969fbf9984d9f19007bf3d8aed701800e5 not found: ID does not exist" Sep 16 13:23:57.201939 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.201909 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng"] Sep 16 13:23:57.208659 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.208641 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:57.210119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.210102 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-15da7b-688d5f8989-58lng"] Sep 16 13:23:57.222414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.222394 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq"] Sep 16 13:23:57.227580 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.227561 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-15da7b-6cfdbcd9cd-7xqbq"] Sep 16 13:23:57.307839 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.307812 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:57.387603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.387519 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309b7e5d-fccc-4a10-975e-e84bbc455796" path="/var/lib/kubelet/pods/309b7e5d-fccc-4a10-975e-e84bbc455796/volumes" Sep 16 13:23:57.387999 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.387978 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbcbd82-0d67-4a36-b089-964b63c3a2d2" path="/var/lib/kubelet/pods/4cbcbd82-0d67-4a36-b089-964b63c3a2d2/volumes" Sep 16 13:23:57.388388 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:57.388371 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fccbedcb-9793-4447-a09b-89ff6c1e6179" path="/var/lib/kubelet/pods/fccbedcb-9793-4447-a09b-89ff6c1e6179/volumes" Sep 16 13:23:58.051294 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:58.051249 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:58.899454 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:58.899420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:23:59.944249 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:23:59.944207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:00.905781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:00.905742 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:01.961263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:01.961228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:02.391905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:02.391813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:02.512038 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:02.512004 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:02.888749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:02.888661 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:02.902527 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:02.902498 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt"] Sep 16 13:24:02.902749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:02.902723 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" podUID="4b236bd1-4b0f-46ff-9640-c93e06caee20" containerName="payload-processing" containerID="cri-o://b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617" gracePeriod=30 Sep 16 13:24:03.021885 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.021855 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:03.137769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.137732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:03.163920 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.163896 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:24:03.168127 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.168104 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg"] Sep 16 13:24:03.174386 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.174365 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9615f1d6-4hxlg"] Sep 16 13:24:03.209747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.209711 2581 generic.go:356] "Generic (PLEG): container finished" podID="4b236bd1-4b0f-46ff-9640-c93e06caee20" containerID="b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617" exitCode=0 Sep 16 13:24:03.209747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.209751 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" Sep 16 13:24:03.209953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.209777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" event={"ID":"4b236bd1-4b0f-46ff-9640-c93e06caee20","Type":"ContainerDied","Data":"b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617"} Sep 16 13:24:03.209953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.209824 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt" event={"ID":"4b236bd1-4b0f-46ff-9640-c93e06caee20","Type":"ContainerDied","Data":"87cbb6404fcd42423232cf3422251bfe8204bb25abdd8ff1d403f0b374baee97"} Sep 16 13:24:03.209953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.209850 2581 scope.go:117] "RemoveContainer" containerID="b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617" Sep 16 13:24:03.218818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.218801 2581 scope.go:117] "RemoveContainer" containerID="b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617" Sep 16 13:24:03.219107 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:24:03.219088 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617\": container with ID starting with b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617 not found: ID does not exist" containerID="b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617" Sep 16 13:24:03.219164 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.219117 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617"} err="failed to get container status \"b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617\": rpc error: code = NotFound desc = could not find container \"b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617\": container with ID starting with b530b109e8ed5d9c2e3020e445e7d91bfe23607340d1bade2e0abc54e5ab0617 not found: ID does not exist" Sep 16 13:24:03.253158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.253126 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:03.268177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.268146 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6fxl\" (UniqueName: \"kubernetes.io/projected/4b236bd1-4b0f-46ff-9640-c93e06caee20-kube-api-access-d6fxl\") pod \"4b236bd1-4b0f-46ff-9640-c93e06caee20\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " Sep 16 13:24:03.268398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.268345 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b236bd1-4b0f-46ff-9640-c93e06caee20-plugins-config-volume\") pod \"4b236bd1-4b0f-46ff-9640-c93e06caee20\" (UID: \"4b236bd1-4b0f-46ff-9640-c93e06caee20\") " Sep 16 13:24:03.268697 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.268673 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b236bd1-4b0f-46ff-9640-c93e06caee20-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b236bd1-4b0f-46ff-9640-c93e06caee20" (UID: "4b236bd1-4b0f-46ff-9640-c93e06caee20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:24:03.268787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.268772 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b236bd1-4b0f-46ff-9640-c93e06caee20-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:24:03.270629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.270575 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b236bd1-4b0f-46ff-9640-c93e06caee20-kube-api-access-d6fxl" (OuterVolumeSpecName: "kube-api-access-d6fxl") pod "4b236bd1-4b0f-46ff-9640-c93e06caee20" (UID: "4b236bd1-4b0f-46ff-9640-c93e06caee20"). InnerVolumeSpecName "kube-api-access-d6fxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:24:03.370044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.370009 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fxl\" (UniqueName: \"kubernetes.io/projected/4b236bd1-4b0f-46ff-9640-c93e06caee20-kube-api-access-d6fxl\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:24:03.383668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.383622 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:03.388411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.388376 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f28702-ad85-4002-8491-7442e22e344d" path="/var/lib/kubelet/pods/42f28702-ad85-4002-8491-7442e22e344d/volumes" Sep 16 13:24:03.531600 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.531569 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt"] Sep 16 13:24:03.540405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:03.540375 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9615f1d6-674c9f8949-7vfwt"] Sep 16 13:24:04.229716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:04.229684 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:04.340939 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:04.340905 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:04.768211 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:04.768175 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:05.391734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:05.391705 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b236bd1-4b0f-46ff-9640-c93e06caee20" path="/var/lib/kubelet/pods/4b236bd1-4b0f-46ff-9640-c93e06caee20/volumes" Sep 16 13:24:05.961355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:05.961321 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:06.966186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:06.966150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:07.837465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:07.837433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:28.162517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:28.162487 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:30.614318 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:30.614273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:36.061575 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:36.061533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:40.748651 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:40.748615 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:41.192637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:41.192533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:41.727246 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:41.727206 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.170762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.170670 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg"] Sep 16 13:24:42.171133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171098 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fccbedcb-9793-4447-a09b-89ff6c1e6179" containerName="payload-processing" Sep 16 13:24:42.171133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171110 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fccbedcb-9793-4447-a09b-89ff6c1e6179" containerName="payload-processing" Sep 16 13:24:42.171133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171122 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f28702-ad85-4002-8491-7442e22e344d" containerName="revoke-keys" Sep 16 13:24:42.171133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171128 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f28702-ad85-4002-8491-7442e22e344d" containerName="revoke-keys" Sep 16 13:24:42.171133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171136 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbcbd82-0d67-4a36-b089-964b63c3a2d2" containerName="payload-pre-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171142 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbcbd82-0d67-4a36-b089-964b63c3a2d2" containerName="payload-pre-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171151 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b236bd1-4b0f-46ff-9640-c93e06caee20" containerName="payload-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171160 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b236bd1-4b0f-46ff-9640-c93e06caee20" containerName="payload-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171227 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b236bd1-4b0f-46ff-9640-c93e06caee20" containerName="payload-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171241 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbcbd82-0d67-4a36-b089-964b63c3a2d2" containerName="payload-pre-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171250 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fccbedcb-9793-4447-a09b-89ff6c1e6179" containerName="payload-processing" Sep 16 13:24:42.171359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.171256 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f28702-ad85-4002-8491-7442e22e344d" containerName="revoke-keys" Sep 16 13:24:42.175759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.175743 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.182729 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.182700 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-588a26-openshift-default-dockercfg-xrvs5\"" Sep 16 13:24:42.213709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-envoy\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.213709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213712 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhspk\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-kube-api-access-qhspk\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.213955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-socket\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.213955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-istio-token\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.213955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-credential-socket\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.213955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-certs\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.213955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a097169-975b-449c-9736-85dad1159c38-istio-podinfo\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.214121 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-data\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.214121 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.213999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a097169-975b-449c-9736-85dad1159c38-istiod-ca-cert\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.221917 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.221884 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.230441 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.230414 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg"] Sep 16 13:24:42.314734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-envoy\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.314734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhspk\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-kube-api-access-qhspk\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-socket\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-istio-token\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-credential-socket\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-certs\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a097169-975b-449c-9736-85dad1159c38-istio-podinfo\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-data\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.314940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a097169-975b-449c-9736-85dad1159c38-istiod-ca-cert\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.315257 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-credential-socket\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.315317 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-certs\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.315342 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-data\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315611 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.315536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-socket\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.315677 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.315657 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a097169-975b-449c-9736-85dad1159c38-istiod-ca-cert\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.316954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.316934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-envoy\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.317397 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.317373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a097169-975b-449c-9736-85dad1159c38-istio-podinfo\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.329939 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.329913 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhspk\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-kube-api-access-qhspk\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.332698 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.332671 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-istio-token\") pod \"e2e-explicit-588a26-openshift-default-8574784b4-gvcsg\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.384872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.384829 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:42.486749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.486720 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:42.779727 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.779695 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg"] Sep 16 13:24:42.780234 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:24:42.780207 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a097169_975b_449c_9736_85dad1159c38.slice/crio-4b4cfbd250a94dc7f5177d65850031987687e8542676b985b0b264fc6d6525f5 WatchSource:0}: Error finding container 4b4cfbd250a94dc7f5177d65850031987687e8542676b985b0b264fc6d6525f5: Status 404 returned error can't find the container with id 4b4cfbd250a94dc7f5177d65850031987687e8542676b985b0b264fc6d6525f5 Sep 16 13:24:42.782483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.782447 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:24:42.782580 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.782518 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:24:42.782580 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:42.782556 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:24:43.440924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:43.440890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" event={"ID":"3a097169-975b-449c-9736-85dad1159c38","Type":"ContainerStarted","Data":"4517023ec4d25c22fa98ffbd9b27444cad5f819721e671dd737c2c800a0e107f"} Sep 16 13:24:43.441330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:43.440937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" event={"ID":"3a097169-975b-449c-9736-85dad1159c38","Type":"ContainerStarted","Data":"4b4cfbd250a94dc7f5177d65850031987687e8542676b985b0b264fc6d6525f5"} Sep 16 13:24:43.471894 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:43.471837 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podStartSLOduration=1.471817787 podStartE2EDuration="1.471817787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:43.46994746 +0000 UTC m=+968.589674029" watchObservedRunningTime="2026-09-16 13:24:43.471817787 +0000 UTC m=+968.591544335" Sep 16 13:24:43.487058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:43.487022 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:43.493921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:43.493899 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:44.340168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:44.340131 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:44.446353 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:44.446324 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:44.447371 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:44.447350 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:24:45.608685 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:45.608645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:45.756831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:45.756792 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:45.916380 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:45.916271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:47.497480 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:47.497439 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:48.950229 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:48.950190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:50.293228 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:50.293194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:51.571908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:51.571871 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:51.759778 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:51.759746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:52.193075 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.193037 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:52.662446 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.662412 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:52.812230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.812194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:24:52.966772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.966743 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb"] Sep 16 13:24:52.970484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.970468 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:52.973607 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.973586 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-588a26-dockercfg-m99tr\"" Sep 16 13:24:52.973702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.973630 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-588a26\"" Sep 16 13:24:52.998934 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:52.998908 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb"] Sep 16 13:24:53.010154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.010116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfw6\" (UniqueName: \"kubernetes.io/projected/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-kube-api-access-7gfw6\") pod \"payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.010363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.010342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.065040 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.065009 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr"] Sep 16 13:24:53.068607 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.068591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.086603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.086580 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr"] Sep 16 13:24:53.111811 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.111779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.111942 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.111826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfw6\" (UniqueName: \"kubernetes.io/projected/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-kube-api-access-7gfw6\") pod \"payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.111942 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.111876 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpgn\" (UniqueName: \"kubernetes.io/projected/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-kube-api-access-bfpgn\") pod \"payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.111942 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.111915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.112606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.112588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.123880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.123854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfw6\" (UniqueName: \"kubernetes.io/projected/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-kube-api-access-7gfw6\") pod \"payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.212694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.212648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpgn\" (UniqueName: \"kubernetes.io/projected/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-kube-api-access-bfpgn\") pod \"payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.212875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.212770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.213462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.213442 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.221790 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.221725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpgn\" (UniqueName: \"kubernetes.io/projected/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-kube-api-access-bfpgn\") pod \"payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.281661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.281634 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:53.378541 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.378507 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:53.561735 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:24:53.561702 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03c55f7_1430_4f6a_9be6_8ed18e9b3b3f.slice/crio-224b4c3d554265f0b3f916b09ea193adb364d9e33d9a3cbc7dc5ac905c6139cd WatchSource:0}: Error finding container 224b4c3d554265f0b3f916b09ea193adb364d9e33d9a3cbc7dc5ac905c6139cd: Status 404 returned error can't find the container with id 224b4c3d554265f0b3f916b09ea193adb364d9e33d9a3cbc7dc5ac905c6139cd Sep 16 13:24:53.561861 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.561759 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb"] Sep 16 13:24:53.656950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:53.656926 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr"] Sep 16 13:24:54.504724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.504690 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" event={"ID":"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e","Type":"ContainerStarted","Data":"4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b"} Sep 16 13:24:54.505210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.504744 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" event={"ID":"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e","Type":"ContainerStarted","Data":"9f98f8d5b79860cfe59840a5c204ff5e58fc65b4ea1fc2c8a13e44623352fad0"} Sep 16 13:24:54.505210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.504774 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:24:54.506324 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.506304 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" event={"ID":"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f","Type":"ContainerStarted","Data":"5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e"} Sep 16 13:24:54.506382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.506343 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" event={"ID":"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f","Type":"ContainerStarted","Data":"224b4c3d554265f0b3f916b09ea193adb364d9e33d9a3cbc7dc5ac905c6139cd"} Sep 16 13:24:54.506425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.506377 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:24:54.524984 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.524931 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" podStartSLOduration=1.524911742 podStartE2EDuration="1.524911742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:54.521873569 +0000 UTC m=+979.641600145" watchObservedRunningTime="2026-09-16 13:24:54.524911742 +0000 UTC m=+979.644638294" Sep 16 13:24:54.541981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:24:54.541936 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" podStartSLOduration=2.541920804 podStartE2EDuration="2.541920804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:54.54061446 +0000 UTC m=+979.660341034" watchObservedRunningTime="2026-09-16 13:24:54.541920804 +0000 UTC m=+979.661647354" Sep 16 13:25:05.515046 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:05.515011 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:25:05.515622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:05.515089 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:25:06.569519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:06.569483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:08.597768 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:08.597684 2581 generic.go:356] "Generic (PLEG): container finished" podID="3a097169-975b-449c-9736-85dad1159c38" containerID="4517023ec4d25c22fa98ffbd9b27444cad5f819721e671dd737c2c800a0e107f" exitCode=0 Sep 16 13:25:08.598182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:08.597767 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" event={"ID":"3a097169-975b-449c-9736-85dad1159c38","Type":"ContainerDied","Data":"4517023ec4d25c22fa98ffbd9b27444cad5f819721e671dd737c2c800a0e107f"} Sep 16 13:25:08.598235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:08.598212 2581 scope.go:117] "RemoveContainer" containerID="4517023ec4d25c22fa98ffbd9b27444cad5f819721e671dd737c2c800a0e107f" Sep 16 13:25:08.599448 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:08.599417 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:25:08.599547 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:08.599486 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:25:08.599547 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:08.599518 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:25:09.605658 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:09.605627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" event={"ID":"3a097169-975b-449c-9736-85dad1159c38","Type":"ContainerStarted","Data":"d45f872e54d8b343a88592a213c54283131c0eeba73fb7e317e758b50c5c6512"} Sep 16 13:25:10.488557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:10.488520 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:25:10.488916 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:10.488891 2581 patch_prober.go:28] interesting pod/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:25:10.489081 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:10.489057 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:25:11.494307 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:11.490614 2581 patch_prober.go:28] interesting pod/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:25:11.494307 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:11.490689 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:25:12.489433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:12.489391 2581 patch_prober.go:28] interesting pod/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:25:12.489650 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:12.489460 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:25:13.487689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:13.487644 2581 patch_prober.go:28] interesting pod/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:25:13.488197 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:13.487716 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:25:14.446524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:14.446487 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:25:14.487716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:14.487677 2581 patch_prober.go:28] interesting pod/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 16 13:25:14.488308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:14.488255 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 16 13:25:15.176298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:15.176242 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:15.521463 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:15.521429 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:25:15.522637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:15.522615 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:25:15.650999 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:15.650964 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:16.321533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:16.321492 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:17.635541 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:17.635502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:19.193661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:19.193613 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:19.757442 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:19.757406 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:20.693913 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:20.693878 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:22.658209 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:22.658178 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:28.267825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:28.267779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:28.496618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:28.496580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:36.209210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:36.209178 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:58.994160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:58.994126 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck"] Sep 16 13:25:58.997712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:58.997692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.001252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.001223 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"openshift-service-ca.crt\"" Sep 16 13:25:59.001252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.001223 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"kube-root-ca.crt\"" Sep 16 13:25:59.001476 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.001310 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"default-dockercfg-2ckdl\"" Sep 16 13:25:59.001476 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.001321 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"e2e-conflict-a-3057f25b-kserve-self-signed-certs\"" Sep 16 13:25:59.009668 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.009643 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck"] Sep 16 13:25:59.031029 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.031001 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:25:59.131239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.131198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd3245-18b9-4528-a934-87f6ffc545a9-tls-certs\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.131420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.131314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-tmp-dir\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.131420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.131376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jc2\" (UniqueName: \"kubernetes.io/projected/c3cd3245-18b9-4528-a934-87f6ffc545a9-kube-api-access-h2jc2\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.131420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.131403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-home\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.131548 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.131426 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-dshm\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.131548 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.131520 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-model-cache\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.232761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.232724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd3245-18b9-4528-a934-87f6ffc545a9-tls-certs\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.232761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.232762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-tmp-dir\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.232972 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.232880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jc2\" (UniqueName: \"kubernetes.io/projected/c3cd3245-18b9-4528-a934-87f6ffc545a9-kube-api-access-h2jc2\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.232972 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.232916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-home\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.232972 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.232948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-dshm\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.233127 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.233020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-model-cache\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.233190 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.233145 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-tmp-dir\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.233248 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.233229 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-home\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.233394 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.233377 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-model-cache\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.235181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.235161 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-dshm\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.235292 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.235253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd3245-18b9-4528-a934-87f6ffc545a9-tls-certs\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.242250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.242225 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jc2\" (UniqueName: \"kubernetes.io/projected/c3cd3245-18b9-4528-a934-87f6ffc545a9-kube-api-access-h2jc2\") pod \"e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.308589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.308515 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:25:59.587709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.587681 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck"] Sep 16 13:25:59.589459 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:25:59.589433 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3cd3245_18b9_4528_a934_87f6ffc545a9.slice/crio-13249c9ddeb12dc9629f2996fd3ecd288d4dccc2d4018283d0ce73a0901e47a5 WatchSource:0}: Error finding container 13249c9ddeb12dc9629f2996fd3ecd288d4dccc2d4018283d0ce73a0901e47a5: Status 404 returned error can't find the container with id 13249c9ddeb12dc9629f2996fd3ecd288d4dccc2d4018283d0ce73a0901e47a5 Sep 16 13:25:59.915160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.915081 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" event={"ID":"c3cd3245-18b9-4528-a934-87f6ffc545a9","Type":"ContainerStarted","Data":"fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949"} Sep 16 13:25:59.915160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.915123 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" event={"ID":"c3cd3245-18b9-4528-a934-87f6ffc545a9","Type":"ContainerStarted","Data":"13249c9ddeb12dc9629f2996fd3ecd288d4dccc2d4018283d0ce73a0901e47a5"} Sep 16 13:25:59.935662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:25:59.935619 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" podStartSLOduration=1.93560437 podStartE2EDuration="1.93560437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:25:59.93333201 +0000 UTC m=+1045.053058657" watchObservedRunningTime="2026-09-16 13:25:59.93560437 +0000 UTC m=+1045.055330918" Sep 16 13:26:01.774403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.774373 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:01.902035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.901994 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6"] Sep 16 13:26:01.905688 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.905670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:01.908276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.908252 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:26:01.908416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.908399 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:26:01.908584 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.908428 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:26:01.915078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:01.915053 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6"] Sep 16 13:26:02.060398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.060276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ff876e-0792-4688-ba32-474d3fa40e8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-588a26-vdmj6\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.060398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.060372 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4k4\" (UniqueName: \"kubernetes.io/projected/45ff876e-0792-4688-ba32-474d3fa40e8d-kube-api-access-dl4k4\") pod \"maas-api-revoke-keys-e2e-explicit-588a26-vdmj6\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.161365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.161333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ff876e-0792-4688-ba32-474d3fa40e8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-588a26-vdmj6\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.161532 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.161386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4k4\" (UniqueName: \"kubernetes.io/projected/45ff876e-0792-4688-ba32-474d3fa40e8d-kube-api-access-dl4k4\") pod \"maas-api-revoke-keys-e2e-explicit-588a26-vdmj6\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.161963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.161943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ff876e-0792-4688-ba32-474d3fa40e8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-588a26-vdmj6\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.170421 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.170400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4k4\" (UniqueName: \"kubernetes.io/projected/45ff876e-0792-4688-ba32-474d3fa40e8d-kube-api-access-dl4k4\") pod \"maas-api-revoke-keys-e2e-explicit-588a26-vdmj6\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.218035 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.218003 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:02.492188 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.492159 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6"] Sep 16 13:26:02.493913 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:26:02.493886 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ff876e_0792_4688_ba32_474d3fa40e8d.slice/crio-106a2ff8dad297128a73b95bfe981334938c23affce032d5ab389c3cf0ebe0b9 WatchSource:0}: Error finding container 106a2ff8dad297128a73b95bfe981334938c23affce032d5ab389c3cf0ebe0b9: Status 404 returned error can't find the container with id 106a2ff8dad297128a73b95bfe981334938c23affce032d5ab389c3cf0ebe0b9 Sep 16 13:26:02.832589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.832486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:02.937031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.936994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" event={"ID":"45ff876e-0792-4688-ba32-474d3fa40e8d","Type":"ContainerStarted","Data":"0399822b65c674c28f8ef0aaad38bd9f02909403f379b957c4e5b20edc2d7bf9"} Sep 16 13:26:02.937208 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.937073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" event={"ID":"45ff876e-0792-4688-ba32-474d3fa40e8d","Type":"ContainerStarted","Data":"106a2ff8dad297128a73b95bfe981334938c23affce032d5ab389c3cf0ebe0b9"} Sep 16 13:26:02.957892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:02.957847 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" podStartSLOduration=1.957832705 podStartE2EDuration="1.957832705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:26:02.957023592 +0000 UTC m=+1048.076750175" watchObservedRunningTime="2026-09-16 13:26:02.957832705 +0000 UTC m=+1048.077559254" Sep 16 13:26:03.819424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:03.819389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:03.945728 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:03.945699 2581 generic.go:356] "Generic (PLEG): container finished" podID="45ff876e-0792-4688-ba32-474d3fa40e8d" containerID="0399822b65c674c28f8ef0aaad38bd9f02909403f379b957c4e5b20edc2d7bf9" exitCode=0 Sep 16 13:26:03.946121 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:03.945797 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" event={"ID":"45ff876e-0792-4688-ba32-474d3fa40e8d","Type":"ContainerDied","Data":"0399822b65c674c28f8ef0aaad38bd9f02909403f379b957c4e5b20edc2d7bf9"} Sep 16 13:26:04.617484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:04.617451 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:05.093552 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.093526 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:05.188918 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.188884 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ff876e-0792-4688-ba32-474d3fa40e8d-maas-api-service-ca\") pod \"45ff876e-0792-4688-ba32-474d3fa40e8d\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " Sep 16 13:26:05.189094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.188924 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl4k4\" (UniqueName: \"kubernetes.io/projected/45ff876e-0792-4688-ba32-474d3fa40e8d-kube-api-access-dl4k4\") pod \"45ff876e-0792-4688-ba32-474d3fa40e8d\" (UID: \"45ff876e-0792-4688-ba32-474d3fa40e8d\") " Sep 16 13:26:05.189336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.189308 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ff876e-0792-4688-ba32-474d3fa40e8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45ff876e-0792-4688-ba32-474d3fa40e8d" (UID: "45ff876e-0792-4688-ba32-474d3fa40e8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:26:05.191063 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.191043 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ff876e-0792-4688-ba32-474d3fa40e8d-kube-api-access-dl4k4" (OuterVolumeSpecName: "kube-api-access-dl4k4") pod "45ff876e-0792-4688-ba32-474d3fa40e8d" (UID: "45ff876e-0792-4688-ba32-474d3fa40e8d"). InnerVolumeSpecName "kube-api-access-dl4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:26:05.289780 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.289709 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ff876e-0792-4688-ba32-474d3fa40e8d-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:05.289780 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.289739 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4k4\" (UniqueName: \"kubernetes.io/projected/45ff876e-0792-4688-ba32-474d3fa40e8d-kube-api-access-dl4k4\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:05.466838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.466802 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:05.958587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.958559 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" Sep 16 13:26:05.958738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.958561 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6" event={"ID":"45ff876e-0792-4688-ba32-474d3fa40e8d","Type":"ContainerDied","Data":"106a2ff8dad297128a73b95bfe981334938c23affce032d5ab389c3cf0ebe0b9"} Sep 16 13:26:05.958738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:05.958680 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106a2ff8dad297128a73b95bfe981334938c23affce032d5ab389c3cf0ebe0b9" Sep 16 13:26:06.177405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:06.177373 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:06.271610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:06.271524 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:06.391335 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:06.391303 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:06.796450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:06.796422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:07.461332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:07.461276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:09.309098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:09.309063 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:26:09.309584 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:09.309111 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:26:09.321651 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:09.321630 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:26:09.994161 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:09.994126 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:26:12.543483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.543447 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:12.593888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.591950 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb"] Sep 16 13:26:12.593888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.592328 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" podUID="f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" containerName="payload-processing" containerID="cri-o://5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e" gracePeriod=30 Sep 16 13:26:12.600257 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.600230 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr"] Sep 16 13:26:12.600703 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.600676 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" podUID="5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" containerName="payload-pre-processing" containerID="cri-o://4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b" gracePeriod=30 Sep 16 13:26:12.630786 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.630756 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:12.734298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.734250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:12.776506 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.776467 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6"] Sep 16 13:26:12.783699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.783436 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-588a26-vdmj6"] Sep 16 13:26:12.828176 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.828147 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:12.861546 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.861523 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:26:12.889025 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.888999 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:26:12.969524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.969481 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfpgn\" (UniqueName: \"kubernetes.io/projected/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-kube-api-access-bfpgn\") pod \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " Sep 16 13:26:12.969687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.969581 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-plugins-config-volume\") pod \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\" (UID: \"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e\") " Sep 16 13:26:12.969687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.969608 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gfw6\" (UniqueName: \"kubernetes.io/projected/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-kube-api-access-7gfw6\") pod \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " Sep 16 13:26:12.969830 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.969704 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-plugins-config-volume\") pod \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\" (UID: \"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f\") " Sep 16 13:26:12.970098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.970067 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" (UID: "5ca1cf92-dee0-4faa-bd1e-829b47dbb09e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:26:12.970211 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.970072 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" (UID: "f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:26:12.971772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.971747 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-kube-api-access-bfpgn" (OuterVolumeSpecName: "kube-api-access-bfpgn") pod "5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" (UID: "5ca1cf92-dee0-4faa-bd1e-829b47dbb09e"). InnerVolumeSpecName "kube-api-access-bfpgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:26:12.971871 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:12.971775 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-kube-api-access-7gfw6" (OuterVolumeSpecName: "kube-api-access-7gfw6") pod "f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" (UID: "f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f"). InnerVolumeSpecName "kube-api-access-7gfw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:26:13.002248 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.002221 2581 generic.go:356] "Generic (PLEG): container finished" podID="5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" containerID="4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b" exitCode=0 Sep 16 13:26:13.002383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.002264 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" Sep 16 13:26:13.002383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.002321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" event={"ID":"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e","Type":"ContainerDied","Data":"4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b"} Sep 16 13:26:13.002383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.002371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr" event={"ID":"5ca1cf92-dee0-4faa-bd1e-829b47dbb09e","Type":"ContainerDied","Data":"9f98f8d5b79860cfe59840a5c204ff5e58fc65b4ea1fc2c8a13e44623352fad0"} Sep 16 13:26:13.002533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.002395 2581 scope.go:117] "RemoveContainer" containerID="4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b" Sep 16 13:26:13.003720 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.003698 2581 generic.go:356] "Generic (PLEG): container finished" podID="f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" containerID="5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e" exitCode=0 Sep 16 13:26:13.003804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.003736 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" event={"ID":"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f","Type":"ContainerDied","Data":"5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e"} Sep 16 13:26:13.003804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.003783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" event={"ID":"f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f","Type":"ContainerDied","Data":"224b4c3d554265f0b3f916b09ea193adb364d9e33d9a3cbc7dc5ac905c6139cd"} Sep 16 13:26:13.003804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.003740 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb" Sep 16 13:26:13.012713 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.012698 2581 scope.go:117] "RemoveContainer" containerID="4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b" Sep 16 13:26:13.012975 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:26:13.012951 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b\": container with ID starting with 4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b not found: ID does not exist" containerID="4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b" Sep 16 13:26:13.013054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.012981 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b"} err="failed to get container status \"4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b\": rpc error: code = NotFound desc = could not find container \"4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b\": container with ID starting with 4299afc572af5d5a5d03157661760a4d13111d09be338dd1bfbc4af6a9ca609b not found: ID does not exist" Sep 16 13:26:13.013054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.012997 2581 scope.go:117] "RemoveContainer" containerID="5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e" Sep 16 13:26:13.021255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.021241 2581 scope.go:117] "RemoveContainer" containerID="5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e" Sep 16 13:26:13.021596 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:26:13.021574 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e\": container with ID starting with 5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e not found: ID does not exist" containerID="5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e" Sep 16 13:26:13.021660 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.021603 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e"} err="failed to get container status \"5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e\": rpc error: code = NotFound desc = could not find container \"5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e\": container with ID starting with 5247d3020dd708cc811886e82e3021fa3ed420ecacb4c105db3cfc6ee5ba1b7e not found: ID does not exist" Sep 16 13:26:13.032076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.032051 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr"] Sep 16 13:26:13.036875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.036853 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-588a26-b699d86f8-q2rgr"] Sep 16 13:26:13.051544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.051516 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb"] Sep 16 13:26:13.053687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.053670 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-588a26-6ffff64f5c-26xzb"] Sep 16 13:26:13.070769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.070719 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:13.070769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.070739 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gfw6\" (UniqueName: \"kubernetes.io/projected/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-kube-api-access-7gfw6\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:13.070769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.070749 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:13.070769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.070758 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpgn\" (UniqueName: \"kubernetes.io/projected/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e-kube-api-access-bfpgn\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:13.177983 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.177952 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:13.275000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.274965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:13.389520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.389436 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ff876e-0792-4688-ba32-474d3fa40e8d" path="/var/lib/kubelet/pods/45ff876e-0792-4688-ba32-474d3fa40e8d/volumes" Sep 16 13:26:13.390010 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.389991 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" path="/var/lib/kubelet/pods/5ca1cf92-dee0-4faa-bd1e-829b47dbb09e/volumes" Sep 16 13:26:13.390466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:13.390449 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" path="/var/lib/kubelet/pods/f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f/volumes" Sep 16 13:26:14.474518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:14.474486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:15.527456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:15.527420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:16.620151 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:16.620113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:17.409500 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:17.409457 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:19.522522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:19.522488 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:19.629466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:19.629429 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:19.843476 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:19.843394 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg"] Sep 16 13:26:19.843673 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:19.843648 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" containerID="cri-o://d45f872e54d8b343a88592a213c54283131c0eeba73fb7e317e758b50c5c6512" gracePeriod=30 Sep 16 13:26:19.865602 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:19.865573 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:20.321398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:20.321370 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:20.715812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:20.715781 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:20.851614 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:20.851576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:22.170774 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:22.170736 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:22.265947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:22.265906 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:22.487952 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:22.487916 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:25.081648 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.081618 2581 generic.go:356] "Generic (PLEG): container finished" podID="3a097169-975b-449c-9736-85dad1159c38" containerID="d45f872e54d8b343a88592a213c54283131c0eeba73fb7e317e758b50c5c6512" exitCode=0 Sep 16 13:26:25.081986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.081672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" event={"ID":"3a097169-975b-449c-9736-85dad1159c38","Type":"ContainerDied","Data":"d45f872e54d8b343a88592a213c54283131c0eeba73fb7e317e758b50c5c6512"} Sep 16 13:26:25.081986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.081730 2581 scope.go:117] "RemoveContainer" containerID="4517023ec4d25c22fa98ffbd9b27444cad5f819721e671dd737c2c800a0e107f" Sep 16 13:26:25.104259 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.104241 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:26:25.189622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189559 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-certs\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.189622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189594 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-socket\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.189805 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189621 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhspk\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-kube-api-access-qhspk\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.189805 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189660 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a097169-975b-449c-9736-85dad1159c38-istio-podinfo\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.189921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189802 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-data\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.189921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189829 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:26:25.189921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189859 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-credential-socket\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.189921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189896 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:26:25.189921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189907 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a097169-975b-449c-9736-85dad1159c38-istiod-ca-cert\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.190160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189948 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-envoy\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.190160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.189986 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-istio-token\") pod \"3a097169-975b-449c-9736-85dad1159c38\" (UID: \"3a097169-975b-449c-9736-85dad1159c38\") " Sep 16 13:26:25.190160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.190060 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:26:25.190344 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.190305 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a097169-975b-449c-9736-85dad1159c38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:26:25.190344 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.190331 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.190525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.190349 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.190525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.190363 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.190762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.190740 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:26:25.192007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.191982 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a097169-975b-449c-9736-85dad1159c38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:26:25.192171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.192146 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-kube-api-access-qhspk" (OuterVolumeSpecName: "kube-api-access-qhspk") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "kube-api-access-qhspk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:26:25.192467 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.192445 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:26:25.192546 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.192473 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a097169-975b-449c-9736-85dad1159c38" (UID: "3a097169-975b-449c-9736-85dad1159c38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:26:25.290974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.290950 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.290974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.290973 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhspk\" (UniqueName: \"kubernetes.io/projected/3a097169-975b-449c-9736-85dad1159c38-kube-api-access-qhspk\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.291130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.290984 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a097169-975b-449c-9736-85dad1159c38-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.291130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.290992 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.291130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.291000 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a097169-975b-449c-9736-85dad1159c38-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:25.291130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:25.291008 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a097169-975b-449c-9736-85dad1159c38-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:26:26.090840 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:26.090812 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" Sep 16 13:26:26.091324 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:26.090844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg" event={"ID":"3a097169-975b-449c-9736-85dad1159c38","Type":"ContainerDied","Data":"4b4cfbd250a94dc7f5177d65850031987687e8542676b985b0b264fc6d6525f5"} Sep 16 13:26:26.091324 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:26.090900 2581 scope.go:117] "RemoveContainer" containerID="d45f872e54d8b343a88592a213c54283131c0eeba73fb7e317e758b50c5c6512" Sep 16 13:26:26.148170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:26.148129 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg"] Sep 16 13:26:26.152538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:26.152503 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-588a26-openshift-default-8574784b4-gvcsg"] Sep 16 13:26:27.387647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:27.387607 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a097169-975b-449c-9736-85dad1159c38" path="/var/lib/kubelet/pods/3a097169-975b-449c-9736-85dad1159c38/volumes" Sep 16 13:26:27.923629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:27.923549 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:29.153969 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:29.153937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:30.801538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:30.801503 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:32.718453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:32.718411 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:36.711520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:36.711488 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:38.992557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:38.992517 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:39.082072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.082033 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:39.206541 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.206489 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:39.904400 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904366 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm"] Sep 16 13:26:39.904782 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904770 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" containerName="payload-processing" Sep 16 13:26:39.904831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904784 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" containerName="payload-processing" Sep 16 13:26:39.904831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904793 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" containerName="payload-pre-processing" Sep 16 13:26:39.904831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904798 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" containerName="payload-pre-processing" Sep 16 13:26:39.904831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904809 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" Sep 16 13:26:39.904831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904815 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904833 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ff876e-0792-4688-ba32-474d3fa40e8d" containerName="revoke-keys" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904838 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ff876e-0792-4688-ba32-474d3fa40e8d" containerName="revoke-keys" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904899 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904905 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ff876e-0792-4688-ba32-474d3fa40e8d" containerName="revoke-keys" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904911 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904920 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03c55f7-1430-4f6a-9be6-8ed18e9b3b3f" containerName="payload-processing" Sep 16 13:26:39.904992 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.904925 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca1cf92-dee0-4faa-bd1e-829b47dbb09e" containerName="payload-pre-processing" Sep 16 13:26:39.909235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.909215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:39.911637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.911617 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"e2e-conflict-b-3057f25b-kserve-self-signed-certs\"" Sep 16 13:26:39.917512 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.917493 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm"] Sep 16 13:26:39.997948 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:39.997897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:40.028879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.028852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-model-cache\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.028996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.028885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tmp-dir\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.028996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.028905 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-home\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.028996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.028921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tls-certs\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.028996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.028969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjqn\" (UniqueName: \"kubernetes.io/projected/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-kube-api-access-6xjqn\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.029167 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.029009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-dshm\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-model-cache\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tmp-dir\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130407 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-home\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130407 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tls-certs\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130407 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjqn\" (UniqueName: \"kubernetes.io/projected/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-kube-api-access-6xjqn\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130407 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-dshm\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130568 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-model-cache\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130582 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-home\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.130756 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.130734 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tmp-dir\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.132528 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.132511 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-dshm\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.132702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.132684 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tls-certs\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.140849 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.140830 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjqn\" (UniqueName: \"kubernetes.io/projected/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-kube-api-access-6xjqn\") pod \"e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.218851 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.218821 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:40.501205 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:26:40.501178 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe2476c1_6b65_41d1_8ff1_2613f1c1fb03.slice/crio-69b3b78467941136504774a02ed18b867ec43ea15a993b54e834ee61788029fc WatchSource:0}: Error finding container 69b3b78467941136504774a02ed18b867ec43ea15a993b54e834ee61788029fc: Status 404 returned error can't find the container with id 69b3b78467941136504774a02ed18b867ec43ea15a993b54e834ee61788029fc Sep 16 13:26:40.501504 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:40.501482 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm"] Sep 16 13:26:41.185125 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:41.185088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" event={"ID":"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03","Type":"ContainerStarted","Data":"ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4"} Sep 16 13:26:41.185527 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:41.185138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" event={"ID":"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03","Type":"ContainerStarted","Data":"69b3b78467941136504774a02ed18b867ec43ea15a993b54e834ee61788029fc"} Sep 16 13:26:41.204597 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:41.204550 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" podStartSLOduration=2.204533421 podStartE2EDuration="2.204533421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:26:41.202956496 +0000 UTC m=+1086.322683044" watchObservedRunningTime="2026-09-16 13:26:41.204533421 +0000 UTC m=+1086.324259969" Sep 16 13:26:49.457666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:49.457628 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:50.219127 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:50.219097 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:50.219335 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:50.219140 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:50.231603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:50.231578 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:50.260209 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:50.260180 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:26:54.919096 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:54.919055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:26:55.138897 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:26:55.138861 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:01.175393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:01.175351 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:01.245652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:01.245618 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:01.360483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:01.360447 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:01.455478 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:01.455450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:11.400000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:11.399963 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:11.471442 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:11.471410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:11.692540 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:11.692456 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:11.819659 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:11.819623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:11.920224 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:11.920189 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:12.745592 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:12.745560 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:12.956687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:12.956658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:14.776786 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:14.776756 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm"] Sep 16 13:27:14.799403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:14.799371 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:15.994979 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:15.994931 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck"] Sep 16 13:27:16.009230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:16.009203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:16.899589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:16.899555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:18.764392 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:18.764362 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:18.827708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:18.827678 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:18.937052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:18.937016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:19.039214 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:19.039138 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:19.137733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:19.137700 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:28.243356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:28.243317 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:29.796218 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:29.796178 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" podUID="fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" containerName="main" containerID="cri-o://ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4" gracePeriod=45 Sep 16 13:27:30.049263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.049197 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:27:30.096984 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.096954 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-home\") pod \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " Sep 16 13:27:30.097166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097006 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tmp-dir\") pod \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " Sep 16 13:27:30.097166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097031 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tls-certs\") pod \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " Sep 16 13:27:30.097166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097056 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-dshm\") pod \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " Sep 16 13:27:30.097166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097075 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjqn\" (UniqueName: \"kubernetes.io/projected/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-kube-api-access-6xjqn\") pod \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " Sep 16 13:27:30.097166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097095 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-model-cache\") pod \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\" (UID: \"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03\") " Sep 16 13:27:30.097491 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097224 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-home" (OuterVolumeSpecName: "home") pod "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" (UID: "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:30.097491 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097309 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" (UID: "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:30.097491 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097400 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-home\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:30.097491 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097417 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tmp-dir\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:30.097491 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.097467 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" (UID: "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:30.099407 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.099382 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-dshm" (OuterVolumeSpecName: "dshm") pod "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" (UID: "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:30.099517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.099441 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" (UID: "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:27:30.099517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.099455 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-kube-api-access-6xjqn" (OuterVolumeSpecName: "kube-api-access-6xjqn") pod "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" (UID: "fe2476c1-6b65-41d1-8ff1-2613f1c1fb03"). InnerVolumeSpecName "kube-api-access-6xjqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:27:30.197816 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.197789 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-tls-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:30.197816 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.197814 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-dshm\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:30.197968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.197824 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjqn\" (UniqueName: \"kubernetes.io/projected/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-kube-api-access-6xjqn\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:30.197968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.197832 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03-model-cache\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:30.492386 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.492354 2581 generic.go:356] "Generic (PLEG): container finished" podID="fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" containerID="ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4" exitCode=0 Sep 16 13:27:30.492576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.492391 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" Sep 16 13:27:30.492576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.492440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" event={"ID":"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03","Type":"ContainerDied","Data":"ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4"} Sep 16 13:27:30.492576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.492485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm" event={"ID":"fe2476c1-6b65-41d1-8ff1-2613f1c1fb03","Type":"ContainerDied","Data":"69b3b78467941136504774a02ed18b867ec43ea15a993b54e834ee61788029fc"} Sep 16 13:27:30.492576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.492507 2581 scope.go:117] "RemoveContainer" containerID="ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4" Sep 16 13:27:30.502634 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.502616 2581 scope.go:117] "RemoveContainer" containerID="ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4" Sep 16 13:27:30.502885 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:27:30.502867 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4\": container with ID starting with ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4 not found: ID does not exist" containerID="ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4" Sep 16 13:27:30.502950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.502896 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4"} err="failed to get container status \"ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4\": rpc error: code = NotFound desc = could not find container \"ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4\": container with ID starting with ff4149e50ce8cb4dd3fb5b6391515f581bac49c98334f8e47ddd816fa3288bd4 not found: ID does not exist" Sep 16 13:27:30.514644 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.514624 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm"] Sep 16 13:27:30.517928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:30.517907 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-b-3057f25b-kserve-5884c955c9-2hkdm"] Sep 16 13:27:31.014103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.014065 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" podUID="c3cd3245-18b9-4528-a934-87f6ffc545a9" containerName="main" containerID="cri-o://fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949" gracePeriod=45 Sep 16 13:27:31.271845 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.271791 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:27:31.305580 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.305557 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2jc2\" (UniqueName: \"kubernetes.io/projected/c3cd3245-18b9-4528-a934-87f6ffc545a9-kube-api-access-h2jc2\") pod \"c3cd3245-18b9-4528-a934-87f6ffc545a9\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " Sep 16 13:27:31.305744 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.305596 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd3245-18b9-4528-a934-87f6ffc545a9-tls-certs\") pod \"c3cd3245-18b9-4528-a934-87f6ffc545a9\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " Sep 16 13:27:31.305744 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.305616 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-dshm\") pod \"c3cd3245-18b9-4528-a934-87f6ffc545a9\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " Sep 16 13:27:31.305744 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.305649 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-home\") pod \"c3cd3245-18b9-4528-a934-87f6ffc545a9\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " Sep 16 13:27:31.305744 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.305717 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-model-cache\") pod \"c3cd3245-18b9-4528-a934-87f6ffc545a9\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " Sep 16 13:27:31.305962 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.305761 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-tmp-dir\") pod \"c3cd3245-18b9-4528-a934-87f6ffc545a9\" (UID: \"c3cd3245-18b9-4528-a934-87f6ffc545a9\") " Sep 16 13:27:31.306054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.306026 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-home" (OuterVolumeSpecName: "home") pod "c3cd3245-18b9-4528-a934-87f6ffc545a9" (UID: "c3cd3245-18b9-4528-a934-87f6ffc545a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:31.306116 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.306044 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3cd3245-18b9-4528-a934-87f6ffc545a9" (UID: "c3cd3245-18b9-4528-a934-87f6ffc545a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:31.306116 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.306072 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3cd3245-18b9-4528-a934-87f6ffc545a9" (UID: "c3cd3245-18b9-4528-a934-87f6ffc545a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:31.307836 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.307797 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-dshm" (OuterVolumeSpecName: "dshm") pod "c3cd3245-18b9-4528-a934-87f6ffc545a9" (UID: "c3cd3245-18b9-4528-a934-87f6ffc545a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:27:31.307974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.307841 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cd3245-18b9-4528-a934-87f6ffc545a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3cd3245-18b9-4528-a934-87f6ffc545a9" (UID: "c3cd3245-18b9-4528-a934-87f6ffc545a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:27:31.307974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.307907 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cd3245-18b9-4528-a934-87f6ffc545a9-kube-api-access-h2jc2" (OuterVolumeSpecName: "kube-api-access-h2jc2") pod "c3cd3245-18b9-4528-a934-87f6ffc545a9" (UID: "c3cd3245-18b9-4528-a934-87f6ffc545a9"). InnerVolumeSpecName "kube-api-access-h2jc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:27:31.387303 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.387259 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" path="/var/lib/kubelet/pods/fe2476c1-6b65-41d1-8ff1-2613f1c1fb03/volumes" Sep 16 13:27:31.406989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.406962 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-model-cache\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:31.406989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.406984 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-tmp-dir\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:31.407133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.406994 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jc2\" (UniqueName: \"kubernetes.io/projected/c3cd3245-18b9-4528-a934-87f6ffc545a9-kube-api-access-h2jc2\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:31.407133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.407013 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3cd3245-18b9-4528-a934-87f6ffc545a9-tls-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:31.407133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.407022 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-dshm\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:31.407133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.407029 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3cd3245-18b9-4528-a934-87f6ffc545a9-home\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:31.499769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.499739 2581 generic.go:356] "Generic (PLEG): container finished" podID="c3cd3245-18b9-4528-a934-87f6ffc545a9" containerID="fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949" exitCode=0 Sep 16 13:27:31.499960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.499783 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" Sep 16 13:27:31.499960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.499827 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" event={"ID":"c3cd3245-18b9-4528-a934-87f6ffc545a9","Type":"ContainerDied","Data":"fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949"} Sep 16 13:27:31.499960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.499867 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck" event={"ID":"c3cd3245-18b9-4528-a934-87f6ffc545a9","Type":"ContainerDied","Data":"13249c9ddeb12dc9629f2996fd3ecd288d4dccc2d4018283d0ce73a0901e47a5"} Sep 16 13:27:31.499960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.499892 2581 scope.go:117] "RemoveContainer" containerID="fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949" Sep 16 13:27:31.509126 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.509109 2581 scope.go:117] "RemoveContainer" containerID="fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949" Sep 16 13:27:31.509417 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:27:31.509396 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949\": container with ID starting with fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949 not found: ID does not exist" containerID="fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949" Sep 16 13:27:31.509472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.509424 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949"} err="failed to get container status \"fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949\": rpc error: code = NotFound desc = could not find container \"fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949\": container with ID starting with fd6481c3fdf355d471a3691f03574b2fe9cc5ee93d58e46fd5fc8a1917deb949 not found: ID does not exist" Sep 16 13:27:31.516915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.516890 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck"] Sep 16 13:27:31.520833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:31.520810 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/e2e-conflict-a-3057f25b-kserve-75c98c6869-npnck"] Sep 16 13:27:33.396637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:33.396599 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3cd3245-18b9-4528-a934-87f6ffc545a9" path="/var/lib/kubelet/pods/c3cd3245-18b9-4528-a934-87f6ffc545a9/volumes" Sep 16 13:27:42.742771 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:42.742718 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:49.245020 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.244981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:49.363256 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363218 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f"] Sep 16 13:27:49.363662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363650 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cd3245-18b9-4528-a934-87f6ffc545a9" containerName="main" Sep 16 13:27:49.363708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363665 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cd3245-18b9-4528-a934-87f6ffc545a9" containerName="main" Sep 16 13:27:49.363708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363673 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" containerName="main" Sep 16 13:27:49.363708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363679 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" containerName="main" Sep 16 13:27:49.363708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363690 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" Sep 16 13:27:49.363708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363696 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a097169-975b-449c-9736-85dad1159c38" containerName="istio-proxy" Sep 16 13:27:49.363863 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363753 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2476c1-6b65-41d1-8ff1-2613f1c1fb03" containerName="main" Sep 16 13:27:49.363863 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.363764 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cd3245-18b9-4528-a934-87f6ffc545a9" containerName="main" Sep 16 13:27:49.368301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.368269 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.371460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.371430 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:27:49.371588 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.371472 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:27:49.372267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.372244 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:27:49.389964 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.389945 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f"] Sep 16 13:27:49.471290 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.471259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2ec955-7ae2-4011-ab47-d15e542da569-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.471472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.471451 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfd2\" (UniqueName: \"kubernetes.io/projected/3c2ec955-7ae2-4011-ab47-d15e542da569-kube-api-access-6tfd2\") pod \"maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.572667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.572588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfd2\" (UniqueName: \"kubernetes.io/projected/3c2ec955-7ae2-4011-ab47-d15e542da569-kube-api-access-6tfd2\") pod \"maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.572667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.572636 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2ec955-7ae2-4011-ab47-d15e542da569-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.573270 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.573250 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2ec955-7ae2-4011-ab47-d15e542da569-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.581648 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.581625 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfd2\" (UniqueName: \"kubernetes.io/projected/3c2ec955-7ae2-4011-ab47-d15e542da569-kube-api-access-6tfd2\") pod \"maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.678968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.678941 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:49.711056 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.711022 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:49.957657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:49.957632 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f"] Sep 16 13:27:49.959318 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:27:49.959290 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2ec955_7ae2_4011_ab47_d15e542da569.slice/crio-08fffe63d4e2dcde808cac39f275f4efb1d0b95218abcb46df08738b9691bb96 WatchSource:0}: Error finding container 08fffe63d4e2dcde808cac39f275f4efb1d0b95218abcb46df08738b9691bb96: Status 404 returned error can't find the container with id 08fffe63d4e2dcde808cac39f275f4efb1d0b95218abcb46df08738b9691bb96 Sep 16 13:27:50.616887 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:50.616793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" event={"ID":"3c2ec955-7ae2-4011-ab47-d15e542da569","Type":"ContainerStarted","Data":"82603e24b45ef72efd4bb72987a2e51fff53b8e5d2ed6a9b78dec764939585e4"} Sep 16 13:27:50.616887 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:50.616873 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" event={"ID":"3c2ec955-7ae2-4011-ab47-d15e542da569","Type":"ContainerStarted","Data":"08fffe63d4e2dcde808cac39f275f4efb1d0b95218abcb46df08738b9691bb96"} Sep 16 13:27:50.635951 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:50.635906 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" podStartSLOduration=1.635892377 podStartE2EDuration="1.635892377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:50.635829545 +0000 UTC m=+1155.755556113" watchObservedRunningTime="2026-09-16 13:27:50.635892377 +0000 UTC m=+1155.755618925" Sep 16 13:27:50.651724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:50.651698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:51.471212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:51.471177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:51.624102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:51.624071 2581 generic.go:356] "Generic (PLEG): container finished" podID="3c2ec955-7ae2-4011-ab47-d15e542da569" containerID="82603e24b45ef72efd4bb72987a2e51fff53b8e5d2ed6a9b78dec764939585e4" exitCode=0 Sep 16 13:27:51.624537 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:51.624119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" event={"ID":"3c2ec955-7ae2-4011-ab47-d15e542da569","Type":"ContainerDied","Data":"82603e24b45ef72efd4bb72987a2e51fff53b8e5d2ed6a9b78dec764939585e4"} Sep 16 13:27:52.315499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:52.315466 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:52.761057 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:52.761035 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:52.904656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:52.904570 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tfd2\" (UniqueName: \"kubernetes.io/projected/3c2ec955-7ae2-4011-ab47-d15e542da569-kube-api-access-6tfd2\") pod \"3c2ec955-7ae2-4011-ab47-d15e542da569\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " Sep 16 13:27:52.904823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:52.904681 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2ec955-7ae2-4011-ab47-d15e542da569-maas-api-service-ca\") pod \"3c2ec955-7ae2-4011-ab47-d15e542da569\" (UID: \"3c2ec955-7ae2-4011-ab47-d15e542da569\") " Sep 16 13:27:52.905077 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:52.905048 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2ec955-7ae2-4011-ab47-d15e542da569-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c2ec955-7ae2-4011-ab47-d15e542da569" (UID: "3c2ec955-7ae2-4011-ab47-d15e542da569"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:27:52.906706 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:52.906681 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2ec955-7ae2-4011-ab47-d15e542da569-kube-api-access-6tfd2" (OuterVolumeSpecName: "kube-api-access-6tfd2") pod "3c2ec955-7ae2-4011-ab47-d15e542da569" (UID: "3c2ec955-7ae2-4011-ab47-d15e542da569"). InnerVolumeSpecName "kube-api-access-6tfd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:27:53.006098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:53.006070 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfd2\" (UniqueName: \"kubernetes.io/projected/3c2ec955-7ae2-4011-ab47-d15e542da569-kube-api-access-6tfd2\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:53.006098 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:53.006096 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c2ec955-7ae2-4011-ab47-d15e542da569-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:27:53.231557 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:53.231521 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:53.637122 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:53.637036 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" event={"ID":"3c2ec955-7ae2-4011-ab47-d15e542da569","Type":"ContainerDied","Data":"08fffe63d4e2dcde808cac39f275f4efb1d0b95218abcb46df08738b9691bb96"} Sep 16 13:27:53.637122 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:53.637091 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08fffe63d4e2dcde808cac39f275f4efb1d0b95218abcb46df08738b9691bb96" Sep 16 13:27:53.637122 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:53.637047 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f" Sep 16 13:27:54.094555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:54.094510 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:27:54.430312 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:27:54.430202 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:28:10.495970 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.495888 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv"] Sep 16 13:28:10.496457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.496276 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2ec955-7ae2-4011-ab47-d15e542da569" containerName="revoke-keys" Sep 16 13:28:10.496457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.496305 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2ec955-7ae2-4011-ab47-d15e542da569" containerName="revoke-keys" Sep 16 13:28:10.496457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.496389 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2ec955-7ae2-4011-ab47-d15e542da569" containerName="revoke-keys" Sep 16 13:28:10.499496 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.499477 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.501995 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.501970 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"default-dockercfg-2ckdl\"" Sep 16 13:28:10.502123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.501973 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"openshift-service-ca.crt\"" Sep 16 13:28:10.502932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.502910 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"kube-root-ca.crt\"" Sep 16 13:28:10.503969 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.503054 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f27ea8\"/\"distinct-2-w4-f27ea8-kserve-self-signed-certs\"" Sep 16 13:28:10.508722 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.508701 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv"] Sep 16 13:28:10.533111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.533088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:28:10.666330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.666266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-home\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.666507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.666390 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-model-cache\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.666507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.666424 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mk9\" (UniqueName: \"kubernetes.io/projected/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-kube-api-access-z2mk9\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.666629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.666543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-dshm\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.666629 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.666591 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tls-certs\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.666736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.666665 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tmp-dir\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-model-cache\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mk9\" (UniqueName: \"kubernetes.io/projected/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-kube-api-access-z2mk9\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768229 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-dshm\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tls-certs\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768443 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tmp-dir\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768443 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-home\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768539 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768523 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-model-cache\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768658 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tmp-dir\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.768801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.768695 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-home\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.770474 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.770451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-dshm\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.770892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.770872 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tls-certs\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.776477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.776456 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mk9\" (UniqueName: \"kubernetes.io/projected/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-kube-api-access-z2mk9\") pod \"distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:10.809994 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:10.809967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:11.084644 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:11.084620 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv"] Sep 16 13:28:11.085648 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:28:11.085621 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90dcbbbf_2b36_47a3_8f81_4b6eb4f3da0f.slice/crio-ee0ddc5e43e3a965a2a5fe109783863f706c17b148cfae697f85c748dd142fc4 WatchSource:0}: Error finding container ee0ddc5e43e3a965a2a5fe109783863f706c17b148cfae697f85c748dd142fc4: Status 404 returned error can't find the container with id ee0ddc5e43e3a965a2a5fe109783863f706c17b148cfae697f85c748dd142fc4 Sep 16 13:28:11.742970 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:11.742940 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" event={"ID":"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f","Type":"ContainerStarted","Data":"fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75"} Sep 16 13:28:11.743560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:11.742981 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" event={"ID":"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f","Type":"ContainerStarted","Data":"ee0ddc5e43e3a965a2a5fe109783863f706c17b148cfae697f85c748dd142fc4"} Sep 16 13:28:11.760846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:11.760801 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" podStartSLOduration=1.760785807 podStartE2EDuration="1.760785807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:28:11.759528949 +0000 UTC m=+1176.879255497" watchObservedRunningTime="2026-09-16 13:28:11.760785807 +0000 UTC m=+1176.880512393" Sep 16 13:28:20.811036 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:20.810997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:20.811534 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:20.811050 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:20.823637 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:20.823612 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:21.815009 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:21.814975 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:28:29.619254 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:29.619185 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:28:35.431901 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:35.431872 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:28:35.434438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:35.434414 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:28:35.508621 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:35.508596 2581 scope.go:117] "RemoveContainer" containerID="6e2437ac676e5805898e1ac6c8be58e1addcc3436d941f16756296ab18eafe5d" Sep 16 13:28:35.517112 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:35.517094 2581 scope.go:117] "RemoveContainer" containerID="e68d632fd3871a0b5a77ec99d6b00cefccd58e8dd2dfdab0528442dd30572f72" Sep 16 13:28:38.240418 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:38.240380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:28:38.296492 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:38.296454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:28:48.957276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:28:48.957219 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:06.809654 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:06.809616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:23.018054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:23.018017 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:28.761021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:28.760937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:28.860853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:28.860822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:29.572574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.572537 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:29.875954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.875875 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd"] Sep 16 13:29:29.879749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.879725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.882140 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.882114 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b97e79df-openshift-default-dockercfg-m7h94\"" Sep 16 13:29:29.892341 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.892320 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd"] Sep 16 13:29:29.910023 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.909988 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:29.984998 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.984962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-envoy\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.984998 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-istio-token\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985258 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-certs\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985258 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-socket\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985258 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b558009-2356-43e2-bcb1-c13e7579966b-istio-podinfo\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985258 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-data\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985578 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b558009-2356-43e2-bcb1-c13e7579966b-istiod-ca-cert\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985578 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-credential-socket\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:29.985578 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:29.985417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wmx\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-kube-api-access-76wmx\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086449 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-data\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b558009-2356-43e2-bcb1-c13e7579966b-istiod-ca-cert\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-credential-socket\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wmx\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-kube-api-access-76wmx\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-envoy\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-istio-token\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-certs\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-socket\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b558009-2356-43e2-bcb1-c13e7579966b-istio-podinfo\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.086954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.086844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-data\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.087174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.087149 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-certs\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.087328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.087304 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b558009-2356-43e2-bcb1-c13e7579966b-istiod-ca-cert\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.087399 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.087354 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-socket\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.087449 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.087430 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-credential-socket\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.089237 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.089207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-envoy\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.089362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.089262 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b558009-2356-43e2-bcb1-c13e7579966b-istio-podinfo\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.095247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.095213 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-istio-token\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.096076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.096049 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wmx\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-kube-api-access-76wmx\") pod \"e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.176372 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.176295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:30.192861 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.192834 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:30.474408 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.474378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd"] Sep 16 13:29:30.475586 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:29:30.475561 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b558009_2356_43e2_bcb1_c13e7579966b.slice/crio-624970c60dc1e7aec1898a6202e9c67d0c5adf4e3ccf7069dd610e25bc5e9971 WatchSource:0}: Error finding container 624970c60dc1e7aec1898a6202e9c67d0c5adf4e3ccf7069dd610e25bc5e9971: Status 404 returned error can't find the container with id 624970c60dc1e7aec1898a6202e9c67d0c5adf4e3ccf7069dd610e25bc5e9971 Sep 16 13:29:30.481394 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.481359 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:29:30.481507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.481442 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:29:30.481507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:30.481487 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:29:31.200955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:31.200920 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" event={"ID":"7b558009-2356-43e2-bcb1-c13e7579966b","Type":"ContainerStarted","Data":"905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61"} Sep 16 13:29:31.201395 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:31.200968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" event={"ID":"7b558009-2356-43e2-bcb1-c13e7579966b","Type":"ContainerStarted","Data":"624970c60dc1e7aec1898a6202e9c67d0c5adf4e3ccf7069dd610e25bc5e9971"} Sep 16 13:29:31.222577 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:31.222535 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podStartSLOduration=2.222520835 podStartE2EDuration="2.222520835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:29:31.220979509 +0000 UTC m=+1256.340706082" watchObservedRunningTime="2026-09-16 13:29:31.222520835 +0000 UTC m=+1256.342247384" Sep 16 13:29:32.193567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:32.193526 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:32.199337 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:32.199310 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:32.206674 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:32.206652 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:32.207528 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:32.207511 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:29:35.336931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:35.336900 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:35.541342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:35.541312 2581 scope.go:117] "RemoveContainer" containerID="d82af04e72837affbd851d67bb73aa260ca4279db79fd4be7e86f79fc197c852" Sep 16 13:29:36.093790 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:36.093753 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:36.247880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:36.247842 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:36.353572 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:36.353473 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:37.353924 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:37.353893 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:38.232510 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:38.232468 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:39.074683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.074653 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:39.174767 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.174729 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:39.551971 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.551933 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:39.649359 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.649317 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:39.731712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.731682 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w"] Sep 16 13:29:39.736348 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.736327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:39.740153 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.740134 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b97e79df-dockercfg-zfxtw\"" Sep 16 13:29:39.740300 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.740140 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b97e79df\"" Sep 16 13:29:39.745451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.745427 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w"] Sep 16 13:29:39.877762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.877679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65201b9-871c-4418-8b36-3e6fb000cc4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:39.877762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.877715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjwd\" (UniqueName: \"kubernetes.io/projected/f65201b9-871c-4418-8b36-3e6fb000cc4c-kube-api-access-hjjwd\") pod \"payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:39.978415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.978384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65201b9-871c-4418-8b36-3e6fb000cc4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:39.978415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.978420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjwd\" (UniqueName: \"kubernetes.io/projected/f65201b9-871c-4418-8b36-3e6fb000cc4c-kube-api-access-hjjwd\") pod \"payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:39.979145 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.979124 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65201b9-871c-4418-8b36-3e6fb000cc4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:39.987423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:39.987398 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjwd\" (UniqueName: \"kubernetes.io/projected/f65201b9-871c-4418-8b36-3e6fb000cc4c-kube-api-access-hjjwd\") pod \"payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:40.048673 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:40.048650 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:40.057803 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:40.057773 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:40.330977 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:40.330949 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w"] Sep 16 13:29:40.333102 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:29:40.333073 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65201b9_871c_4418_8b36_3e6fb000cc4c.slice/crio-81b65e5b229c78dae5f8e351cfccbef531fce0190acc7703082364651cc31ad3 WatchSource:0}: Error finding container 81b65e5b229c78dae5f8e351cfccbef531fce0190acc7703082364651cc31ad3: Status 404 returned error can't find the container with id 81b65e5b229c78dae5f8e351cfccbef531fce0190acc7703082364651cc31ad3 Sep 16 13:29:40.823657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:40.823620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:41.258061 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:41.258026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" event={"ID":"f65201b9-871c-4418-8b36-3e6fb000cc4c","Type":"ContainerStarted","Data":"0436ca2e86eff6b9be7ae89bf6fe35da7fc0ab87c162b6c070e4a7358f0a3606"} Sep 16 13:29:41.258244 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:41.258078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" event={"ID":"f65201b9-871c-4418-8b36-3e6fb000cc4c","Type":"ContainerStarted","Data":"81b65e5b229c78dae5f8e351cfccbef531fce0190acc7703082364651cc31ad3"} Sep 16 13:29:41.258244 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:41.258106 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:41.276040 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:41.275993 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" podStartSLOduration=2.27597623 podStartE2EDuration="2.27597623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:29:41.274202047 +0000 UTC m=+1266.393928596" watchObservedRunningTime="2026-09-16 13:29:41.27597623 +0000 UTC m=+1266.395702776" Sep 16 13:29:41.733597 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:41.733563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:29:52.265853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:52.265824 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:29:59.373119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:59.373085 2581 generic.go:356] "Generic (PLEG): container finished" podID="7b558009-2356-43e2-bcb1-c13e7579966b" containerID="905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61" exitCode=0 Sep 16 13:29:59.373660 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:59.373166 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" event={"ID":"7b558009-2356-43e2-bcb1-c13e7579966b","Type":"ContainerDied","Data":"905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61"} Sep 16 13:29:59.373761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:59.373743 2581 scope.go:117] "RemoveContainer" containerID="905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61" Sep 16 13:29:59.374843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:59.374817 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:29:59.374899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:59.374883 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:29:59.374935 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:29:59.374915 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:30:00.137154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.137118 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns"] Sep 16 13:30:00.140865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.140847 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:00.143579 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.143553 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:30:00.143724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.143641 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:30:00.144428 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.144413 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:30:00.145264 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.145240 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5"] Sep 16 13:30:00.148759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.148743 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:00.149615 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.149580 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs"] Sep 16 13:30:00.152948 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.152929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:00.155860 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.155810 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns"] Sep 16 13:30:00.160469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.160094 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5"] Sep 16 13:30:00.171819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.171796 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7"] Sep 16 13:30:00.175871 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.175853 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs"] Sep 16 13:30:00.175985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.175951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:00.194387 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.194362 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7"] Sep 16 13:30:00.194893 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.194809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54fzn\" (UniqueName: \"kubernetes.io/projected/22a4d87f-c263-4b9c-b18b-361a4dab14a7-kube-api-access-54fzn\") pod \"maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs\" (UID: \"22a4d87f-c263-4b9c-b18b-361a4dab14a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:00.194893 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.194860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxr84\" (UniqueName: \"kubernetes.io/projected/3fc99f3b-6a17-4f8d-8158-cd2c463031c6-kube-api-access-jxr84\") pod \"maas-api-key-cleanup-29826090-vljns\" (UID: \"3fc99f3b-6a17-4f8d-8158-cd2c463031c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:00.195066 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.194938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrh6\" (UniqueName: \"kubernetes.io/projected/5aea6ca3-24ff-46eb-8947-e8827f0ca741-kube-api-access-gkrh6\") pod \"maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5\" (UID: \"5aea6ca3-24ff-46eb-8947-e8827f0ca741\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:00.295736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.295690 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbfp\" (UniqueName: \"kubernetes.io/projected/1dfc26f0-2732-4a54-a254-0e75618cbf04-kube-api-access-mrbfp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7\" (UID: \"1dfc26f0-2732-4a54-a254-0e75618cbf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:00.296088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.296066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54fzn\" (UniqueName: \"kubernetes.io/projected/22a4d87f-c263-4b9c-b18b-361a4dab14a7-kube-api-access-54fzn\") pod \"maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs\" (UID: \"22a4d87f-c263-4b9c-b18b-361a4dab14a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:00.296601 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.296576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxr84\" (UniqueName: \"kubernetes.io/projected/3fc99f3b-6a17-4f8d-8158-cd2c463031c6-kube-api-access-jxr84\") pod \"maas-api-key-cleanup-29826090-vljns\" (UID: \"3fc99f3b-6a17-4f8d-8158-cd2c463031c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:00.297052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.297032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrh6\" (UniqueName: \"kubernetes.io/projected/5aea6ca3-24ff-46eb-8947-e8827f0ca741-kube-api-access-gkrh6\") pod \"maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5\" (UID: \"5aea6ca3-24ff-46eb-8947-e8827f0ca741\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:00.308074 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.308012 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54fzn\" (UniqueName: \"kubernetes.io/projected/22a4d87f-c263-4b9c-b18b-361a4dab14a7-kube-api-access-54fzn\") pod \"maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs\" (UID: \"22a4d87f-c263-4b9c-b18b-361a4dab14a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:00.308932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.308906 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxr84\" (UniqueName: \"kubernetes.io/projected/3fc99f3b-6a17-4f8d-8158-cd2c463031c6-kube-api-access-jxr84\") pod \"maas-api-key-cleanup-29826090-vljns\" (UID: \"3fc99f3b-6a17-4f8d-8158-cd2c463031c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:00.310188 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.310160 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrh6\" (UniqueName: \"kubernetes.io/projected/5aea6ca3-24ff-46eb-8947-e8827f0ca741-kube-api-access-gkrh6\") pod \"maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5\" (UID: \"5aea6ca3-24ff-46eb-8947-e8827f0ca741\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:00.381077 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.381044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" event={"ID":"7b558009-2356-43e2-bcb1-c13e7579966b","Type":"ContainerStarted","Data":"aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88"} Sep 16 13:30:00.398252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.398151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbfp\" (UniqueName: \"kubernetes.io/projected/1dfc26f0-2732-4a54-a254-0e75618cbf04-kube-api-access-mrbfp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7\" (UID: \"1dfc26f0-2732-4a54-a254-0e75618cbf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:00.408005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.407965 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbfp\" (UniqueName: \"kubernetes.io/projected/1dfc26f0-2732-4a54-a254-0e75618cbf04-kube-api-access-mrbfp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7\" (UID: \"1dfc26f0-2732-4a54-a254-0e75618cbf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:00.452298 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.452254 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:00.465642 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.465612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:00.474443 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.474420 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:00.491740 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.490415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:00.756143 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.756116 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns"] Sep 16 13:30:00.756644 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:30:00.756618 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc99f3b_6a17_4f8d_8158_cd2c463031c6.slice/crio-439b5874269255ceb343d29346f42308077fc56263e0a453a2da37c2e04116e9 WatchSource:0}: Error finding container 439b5874269255ceb343d29346f42308077fc56263e0a453a2da37c2e04116e9: Status 404 returned error can't find the container with id 439b5874269255ceb343d29346f42308077fc56263e0a453a2da37c2e04116e9 Sep 16 13:30:00.797306 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.791166 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5"] Sep 16 13:30:00.838958 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.838935 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7"] Sep 16 13:30:00.842883 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:00.842863 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs"] Sep 16 13:30:00.854572 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:30:00.854549 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfc26f0_2732_4a54_a254_0e75618cbf04.slice/crio-04c378d04120ebaccc1d93f4d85c92df9de5ee36e733d38c793a65e3a05fa9f9 WatchSource:0}: Error finding container 04c378d04120ebaccc1d93f4d85c92df9de5ee36e733d38c793a65e3a05fa9f9: Status 404 returned error can't find the container with id 04c378d04120ebaccc1d93f4d85c92df9de5ee36e733d38c793a65e3a05fa9f9 Sep 16 13:30:00.854821 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:30:00.854796 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a4d87f_c263_4b9c_b18b_361a4dab14a7.slice/crio-2030b8c706cd029136fecee825d97d343e0657154eea5df5b53bf7f77d394d48 WatchSource:0}: Error finding container 2030b8c706cd029136fecee825d97d343e0657154eea5df5b53bf7f77d394d48: Status 404 returned error can't find the container with id 2030b8c706cd029136fecee825d97d343e0657154eea5df5b53bf7f77d394d48 Sep 16 13:30:01.197327 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.195498 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:30:01.197327 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.195658 2581 patch_prober.go:28] interesting pod/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 16 13:30:01.197327 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.195703 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 16 13:30:01.235703 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.235677 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:01.357249 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.357159 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:01.395094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.395056 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" event={"ID":"5aea6ca3-24ff-46eb-8947-e8827f0ca741","Type":"ContainerStarted","Data":"1e7a7722fad9bc12596a68c9190b36c6d32ec97c30c1ea126f0f4ee66e60422e"} Sep 16 13:30:01.395652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.395627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" event={"ID":"5aea6ca3-24ff-46eb-8947-e8827f0ca741","Type":"ContainerStarted","Data":"42f06de594d5116c170a0acf3f9e141942b849c2cec8979aa1aec88c953b448a"} Sep 16 13:30:01.398678 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.398631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" event={"ID":"22a4d87f-c263-4b9c-b18b-361a4dab14a7","Type":"ContainerStarted","Data":"d984697ab426cb2ae9e9d704ad293144580511784a20316b35149a8def7d98f3"} Sep 16 13:30:01.398820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.398688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" event={"ID":"22a4d87f-c263-4b9c-b18b-361a4dab14a7","Type":"ContainerStarted","Data":"2030b8c706cd029136fecee825d97d343e0657154eea5df5b53bf7f77d394d48"} Sep 16 13:30:01.401238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.401210 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" event={"ID":"1dfc26f0-2732-4a54-a254-0e75618cbf04","Type":"ContainerStarted","Data":"61845b6eec64cd3a67051f23f6c30758b1dfc7482c8e95ab539e156484a248b4"} Sep 16 13:30:01.401361 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.401262 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" event={"ID":"1dfc26f0-2732-4a54-a254-0e75618cbf04","Type":"ContainerStarted","Data":"04c378d04120ebaccc1d93f4d85c92df9de5ee36e733d38c793a65e3a05fa9f9"} Sep 16 13:30:01.404031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.403994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" event={"ID":"3fc99f3b-6a17-4f8d-8158-cd2c463031c6","Type":"ContainerStarted","Data":"e533a903ec9fc1b714834530609401d2eeadc976f0c6e1190e129dee99316f6a"} Sep 16 13:30:01.404136 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.404041 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" event={"ID":"3fc99f3b-6a17-4f8d-8158-cd2c463031c6","Type":"ContainerStarted","Data":"439b5874269255ceb343d29346f42308077fc56263e0a453a2da37c2e04116e9"} Sep 16 13:30:01.418158 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.418112 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" podStartSLOduration=1.41809822 podStartE2EDuration="1.41809822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:01.412362181 +0000 UTC m=+1286.532088755" watchObservedRunningTime="2026-09-16 13:30:01.41809822 +0000 UTC m=+1286.537824768" Sep 16 13:30:01.431840 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.431796 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" podStartSLOduration=1.431782925 podStartE2EDuration="1.431782925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:01.430482242 +0000 UTC m=+1286.550208802" watchObservedRunningTime="2026-09-16 13:30:01.431782925 +0000 UTC m=+1286.551509473" Sep 16 13:30:01.447375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.447332 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" podStartSLOduration=1.4473188160000001 podStartE2EDuration="1.447318816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:01.445861707 +0000 UTC m=+1286.565588273" watchObservedRunningTime="2026-09-16 13:30:01.447318816 +0000 UTC m=+1286.567045358" Sep 16 13:30:01.466854 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:01.466813 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" podStartSLOduration=1.4667997910000001 podStartE2EDuration="1.466799791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:01.464178544 +0000 UTC m=+1286.583905110" watchObservedRunningTime="2026-09-16 13:30:01.466799791 +0000 UTC m=+1286.586526339" Sep 16 13:30:02.043365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.043328 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:02.146518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.146487 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:02.193736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.193702 2581 patch_prober.go:28] interesting pod/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 16 13:30:02.193900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.193777 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 16 13:30:02.206930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.206898 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:30:02.313152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.313086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:02.415931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.415899 2581 generic.go:356] "Generic (PLEG): container finished" podID="5aea6ca3-24ff-46eb-8947-e8827f0ca741" containerID="1e7a7722fad9bc12596a68c9190b36c6d32ec97c30c1ea126f0f4ee66e60422e" exitCode=0 Sep 16 13:30:02.416328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.416015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" event={"ID":"5aea6ca3-24ff-46eb-8947-e8827f0ca741","Type":"ContainerDied","Data":"1e7a7722fad9bc12596a68c9190b36c6d32ec97c30c1ea126f0f4ee66e60422e"} Sep 16 13:30:02.417999 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.417976 2581 generic.go:356] "Generic (PLEG): container finished" podID="22a4d87f-c263-4b9c-b18b-361a4dab14a7" containerID="d984697ab426cb2ae9e9d704ad293144580511784a20316b35149a8def7d98f3" exitCode=0 Sep 16 13:30:02.418101 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.418052 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" event={"ID":"22a4d87f-c263-4b9c-b18b-361a4dab14a7","Type":"ContainerDied","Data":"d984697ab426cb2ae9e9d704ad293144580511784a20316b35149a8def7d98f3"} Sep 16 13:30:02.419776 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.419756 2581 generic.go:356] "Generic (PLEG): container finished" podID="1dfc26f0-2732-4a54-a254-0e75618cbf04" containerID="61845b6eec64cd3a67051f23f6c30758b1dfc7482c8e95ab539e156484a248b4" exitCode=0 Sep 16 13:30:02.419879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.419863 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" event={"ID":"1dfc26f0-2732-4a54-a254-0e75618cbf04","Type":"ContainerDied","Data":"61845b6eec64cd3a67051f23f6c30758b1dfc7482c8e95ab539e156484a248b4"} Sep 16 13:30:02.421296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.421265 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fc99f3b-6a17-4f8d-8158-cd2c463031c6" containerID="e533a903ec9fc1b714834530609401d2eeadc976f0c6e1190e129dee99316f6a" exitCode=0 Sep 16 13:30:02.421383 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.421316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" event={"ID":"3fc99f3b-6a17-4f8d-8158-cd2c463031c6","Type":"ContainerDied","Data":"e533a903ec9fc1b714834530609401d2eeadc976f0c6e1190e129dee99316f6a"} Sep 16 13:30:02.460043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.460002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:02.929634 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:02.929601 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:03.194215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.194107 2581 patch_prober.go:28] interesting pod/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 16 13:30:03.194215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.194182 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 16 13:30:03.628735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.627602 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:03.715948 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.715596 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:03.739798 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.739768 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:03.742416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.742394 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:03.745455 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.745430 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkrh6\" (UniqueName: \"kubernetes.io/projected/5aea6ca3-24ff-46eb-8947-e8827f0ca741-kube-api-access-gkrh6\") pod \"5aea6ca3-24ff-46eb-8947-e8827f0ca741\" (UID: \"5aea6ca3-24ff-46eb-8947-e8827f0ca741\") " Sep 16 13:30:03.747751 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.747697 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aea6ca3-24ff-46eb-8947-e8827f0ca741-kube-api-access-gkrh6" (OuterVolumeSpecName: "kube-api-access-gkrh6") pod "5aea6ca3-24ff-46eb-8947-e8827f0ca741" (UID: "5aea6ca3-24ff-46eb-8947-e8827f0ca741"). InnerVolumeSpecName "kube-api-access-gkrh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:03.846231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.846198 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrbfp\" (UniqueName: \"kubernetes.io/projected/1dfc26f0-2732-4a54-a254-0e75618cbf04-kube-api-access-mrbfp\") pod \"1dfc26f0-2732-4a54-a254-0e75618cbf04\" (UID: \"1dfc26f0-2732-4a54-a254-0e75618cbf04\") " Sep 16 13:30:03.846431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.846337 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxr84\" (UniqueName: \"kubernetes.io/projected/3fc99f3b-6a17-4f8d-8158-cd2c463031c6-kube-api-access-jxr84\") pod \"3fc99f3b-6a17-4f8d-8158-cd2c463031c6\" (UID: \"3fc99f3b-6a17-4f8d-8158-cd2c463031c6\") " Sep 16 13:30:03.846431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.846370 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54fzn\" (UniqueName: \"kubernetes.io/projected/22a4d87f-c263-4b9c-b18b-361a4dab14a7-kube-api-access-54fzn\") pod \"22a4d87f-c263-4b9c-b18b-361a4dab14a7\" (UID: \"22a4d87f-c263-4b9c-b18b-361a4dab14a7\") " Sep 16 13:30:03.846748 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.846705 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkrh6\" (UniqueName: \"kubernetes.io/projected/5aea6ca3-24ff-46eb-8947-e8827f0ca741-kube-api-access-gkrh6\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:03.847708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.847680 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:03.848726 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.848701 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfc26f0-2732-4a54-a254-0e75618cbf04-kube-api-access-mrbfp" (OuterVolumeSpecName: "kube-api-access-mrbfp") pod "1dfc26f0-2732-4a54-a254-0e75618cbf04" (UID: "1dfc26f0-2732-4a54-a254-0e75618cbf04"). InnerVolumeSpecName "kube-api-access-mrbfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:03.849175 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.849152 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc99f3b-6a17-4f8d-8158-cd2c463031c6-kube-api-access-jxr84" (OuterVolumeSpecName: "kube-api-access-jxr84") pod "3fc99f3b-6a17-4f8d-8158-cd2c463031c6" (UID: "3fc99f3b-6a17-4f8d-8158-cd2c463031c6"). InnerVolumeSpecName "kube-api-access-jxr84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:03.849787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.849769 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a4d87f-c263-4b9c-b18b-361a4dab14a7-kube-api-access-54fzn" (OuterVolumeSpecName: "kube-api-access-54fzn") pod "22a4d87f-c263-4b9c-b18b-361a4dab14a7" (UID: "22a4d87f-c263-4b9c-b18b-361a4dab14a7"). InnerVolumeSpecName "kube-api-access-54fzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:03.947561 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.947524 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54fzn\" (UniqueName: \"kubernetes.io/projected/22a4d87f-c263-4b9c-b18b-361a4dab14a7-kube-api-access-54fzn\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:03.947561 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.947556 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbfp\" (UniqueName: \"kubernetes.io/projected/1dfc26f0-2732-4a54-a254-0e75618cbf04-kube-api-access-mrbfp\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:03.947561 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:03.947566 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxr84\" (UniqueName: \"kubernetes.io/projected/3fc99f3b-6a17-4f8d-8158-cd2c463031c6-kube-api-access-jxr84\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:04.193272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.193176 2581 patch_prober.go:28] interesting pod/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 16 13:30:04.193272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.193246 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 16 13:30:04.434800 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.434774 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" Sep 16 13:30:04.434988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.434813 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5" event={"ID":"5aea6ca3-24ff-46eb-8947-e8827f0ca741","Type":"ContainerDied","Data":"42f06de594d5116c170a0acf3f9e141942b849c2cec8979aa1aec88c953b448a"} Sep 16 13:30:04.434988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.434862 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f06de594d5116c170a0acf3f9e141942b849c2cec8979aa1aec88c953b448a" Sep 16 13:30:04.436331 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.436304 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" Sep 16 13:30:04.436479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.436325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs" event={"ID":"22a4d87f-c263-4b9c-b18b-361a4dab14a7","Type":"ContainerDied","Data":"2030b8c706cd029136fecee825d97d343e0657154eea5df5b53bf7f77d394d48"} Sep 16 13:30:04.436479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.436365 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2030b8c706cd029136fecee825d97d343e0657154eea5df5b53bf7f77d394d48" Sep 16 13:30:04.437678 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.437652 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" Sep 16 13:30:04.438036 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.437974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7" event={"ID":"1dfc26f0-2732-4a54-a254-0e75618cbf04","Type":"ContainerDied","Data":"04c378d04120ebaccc1d93f4d85c92df9de5ee36e733d38c793a65e3a05fa9f9"} Sep 16 13:30:04.438111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.438059 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c378d04120ebaccc1d93f4d85c92df9de5ee36e733d38c793a65e3a05fa9f9" Sep 16 13:30:04.447273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.447183 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" event={"ID":"3fc99f3b-6a17-4f8d-8158-cd2c463031c6","Type":"ContainerDied","Data":"439b5874269255ceb343d29346f42308077fc56263e0a453a2da37c2e04116e9"} Sep 16 13:30:04.447433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.447275 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="439b5874269255ceb343d29346f42308077fc56263e0a453a2da37c2e04116e9" Sep 16 13:30:04.447433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.447381 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826090-vljns" Sep 16 13:30:04.577344 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:04.577309 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:05.194046 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:05.193999 2581 patch_prober.go:28] interesting pod/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Sep 16 13:30:05.194595 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:05.194083 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Sep 16 13:30:06.198094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:06.198059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:30:06.199216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:06.199189 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:30:08.426324 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.426265 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:08.526706 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.526673 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:08.801878 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.801841 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f"] Sep 16 13:30:08.802252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802240 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a4d87f-c263-4b9c-b18b-361a4dab14a7" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802254 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a4d87f-c263-4b9c-b18b-361a4dab14a7" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802294 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dfc26f0-2732-4a54-a254-0e75618cbf04" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802301 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dfc26f0-2732-4a54-a254-0e75618cbf04" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802308 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aea6ca3-24ff-46eb-8947-e8827f0ca741" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802313 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aea6ca3-24ff-46eb-8947-e8827f0ca741" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802323 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc99f3b-6a17-4f8d-8158-cd2c463031c6" containerName="cleanup" Sep 16 13:30:08.802332 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802328 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc99f3b-6a17-4f8d-8158-cd2c463031c6" containerName="cleanup" Sep 16 13:30:08.802582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802399 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc99f3b-6a17-4f8d-8158-cd2c463031c6" containerName="cleanup" Sep 16 13:30:08.802582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802407 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dfc26f0-2732-4a54-a254-0e75618cbf04" containerName="cleanup" Sep 16 13:30:08.802582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802413 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aea6ca3-24ff-46eb-8947-e8827f0ca741" containerName="cleanup" Sep 16 13:30:08.802582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.802419 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a4d87f-c263-4b9c-b18b-361a4dab14a7" containerName="cleanup" Sep 16 13:30:08.805436 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.805418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:08.808342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.808319 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:30:08.808463 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.808343 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:30:08.809524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.809508 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:30:08.824367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.821307 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f"] Sep 16 13:30:08.893249 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.893212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbfg\" (UniqueName: \"kubernetes.io/projected/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-kube-api-access-xgbfg\") pod \"maas-api-revoke-keys-e2e-mt-b97e79df-q958f\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:08.893425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.893360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b97e79df-q958f\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:08.994221 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.994193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b97e79df-q958f\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:08.994426 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.994254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbfg\" (UniqueName: \"kubernetes.io/projected/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-kube-api-access-xgbfg\") pod \"maas-api-revoke-keys-e2e-mt-b97e79df-q958f\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:08.994812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:08.994791 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b97e79df-q958f\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:09.002411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:09.002391 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbfg\" (UniqueName: \"kubernetes.io/projected/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-kube-api-access-xgbfg\") pod \"maas-api-revoke-keys-e2e-mt-b97e79df-q958f\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:09.117825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:09.117739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:09.391265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:09.391240 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f"] Sep 16 13:30:09.392911 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:30:09.392885 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcca3b2_8e48_48dd_964d_69fbc9b91cea.slice/crio-5d69774c3a87e05feab80627978cd3e3d1fb5129d3c1de2e08ff30642c77fc4d WatchSource:0}: Error finding container 5d69774c3a87e05feab80627978cd3e3d1fb5129d3c1de2e08ff30642c77fc4d: Status 404 returned error can't find the container with id 5d69774c3a87e05feab80627978cd3e3d1fb5129d3c1de2e08ff30642c77fc4d Sep 16 13:30:09.482960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:09.482906 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" event={"ID":"4fcca3b2-8e48-48dd-964d-69fbc9b91cea","Type":"ContainerStarted","Data":"5d69774c3a87e05feab80627978cd3e3d1fb5129d3c1de2e08ff30642c77fc4d"} Sep 16 13:30:09.747646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:09.747611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:10.423643 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:10.423607 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:10.495386 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:10.495353 2581 generic.go:356] "Generic (PLEG): container finished" podID="4fcca3b2-8e48-48dd-964d-69fbc9b91cea" containerID="94efe9d177d0f396157f34b5288ed94cbbee842ac38406fc88b906b2973744ec" exitCode=0 Sep 16 13:30:10.496450 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:10.496203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" event={"ID":"4fcca3b2-8e48-48dd-964d-69fbc9b91cea","Type":"ContainerDied","Data":"94efe9d177d0f396157f34b5288ed94cbbee842ac38406fc88b906b2973744ec"} Sep 16 13:30:11.114712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.114679 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:11.634986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.634965 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:11.720095 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.720062 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-maas-api-service-ca\") pod \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " Sep 16 13:30:11.720252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.720242 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgbfg\" (UniqueName: \"kubernetes.io/projected/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-kube-api-access-xgbfg\") pod \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\" (UID: \"4fcca3b2-8e48-48dd-964d-69fbc9b91cea\") " Sep 16 13:30:11.720488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.720462 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fcca3b2-8e48-48dd-964d-69fbc9b91cea" (UID: "4fcca3b2-8e48-48dd-964d-69fbc9b91cea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:30:11.722266 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.722239 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-kube-api-access-xgbfg" (OuterVolumeSpecName: "kube-api-access-xgbfg") pod "4fcca3b2-8e48-48dd-964d-69fbc9b91cea" (UID: "4fcca3b2-8e48-48dd-964d-69fbc9b91cea"). InnerVolumeSpecName "kube-api-access-xgbfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:11.821869 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.821804 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbfg\" (UniqueName: \"kubernetes.io/projected/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-kube-api-access-xgbfg\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:11.821869 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.821827 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcca3b2-8e48-48dd-964d-69fbc9b91cea-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:11.851367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:11.851334 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:12.511827 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:12.511796 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" Sep 16 13:30:12.512007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:12.511836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f" event={"ID":"4fcca3b2-8e48-48dd-964d-69fbc9b91cea","Type":"ContainerDied","Data":"5d69774c3a87e05feab80627978cd3e3d1fb5129d3c1de2e08ff30642c77fc4d"} Sep 16 13:30:12.512007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:12.511882 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d69774c3a87e05feab80627978cd3e3d1fb5129d3c1de2e08ff30642c77fc4d" Sep 16 13:30:12.535960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:12.535937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:19.343453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.343418 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w"] Sep 16 13:30:19.343842 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.343721 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" podUID="f65201b9-871c-4418-8b36-3e6fb000cc4c" containerName="payload-pre-processing" containerID="cri-o://0436ca2e86eff6b9be7ae89bf6fe35da7fc0ab87c162b6c070e4a7358f0a3606" gracePeriod=30 Sep 16 13:30:19.415442 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.415393 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:19.513636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.513606 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f"] Sep 16 13:30:19.517378 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.517352 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b97e79df-q958f"] Sep 16 13:30:19.525000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.524968 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:19.560597 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.560567 2581 generic.go:356] "Generic (PLEG): container finished" podID="f65201b9-871c-4418-8b36-3e6fb000cc4c" containerID="0436ca2e86eff6b9be7ae89bf6fe35da7fc0ab87c162b6c070e4a7358f0a3606" exitCode=0 Sep 16 13:30:19.560759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.560641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" event={"ID":"f65201b9-871c-4418-8b36-3e6fb000cc4c","Type":"ContainerDied","Data":"0436ca2e86eff6b9be7ae89bf6fe35da7fc0ab87c162b6c070e4a7358f0a3606"} Sep 16 13:30:19.600764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.600706 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:30:19.611570 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.611544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:19.695099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.695065 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65201b9-871c-4418-8b36-3e6fb000cc4c-plugins-config-volume\") pod \"f65201b9-871c-4418-8b36-3e6fb000cc4c\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " Sep 16 13:30:19.695321 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.695118 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjjwd\" (UniqueName: \"kubernetes.io/projected/f65201b9-871c-4418-8b36-3e6fb000cc4c-kube-api-access-hjjwd\") pod \"f65201b9-871c-4418-8b36-3e6fb000cc4c\" (UID: \"f65201b9-871c-4418-8b36-3e6fb000cc4c\") " Sep 16 13:30:19.695857 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.695801 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65201b9-871c-4418-8b36-3e6fb000cc4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f65201b9-871c-4418-8b36-3e6fb000cc4c" (UID: "f65201b9-871c-4418-8b36-3e6fb000cc4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:30:19.698460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.698429 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65201b9-871c-4418-8b36-3e6fb000cc4c-kube-api-access-hjjwd" (OuterVolumeSpecName: "kube-api-access-hjjwd") pod "f65201b9-871c-4418-8b36-3e6fb000cc4c" (UID: "f65201b9-871c-4418-8b36-3e6fb000cc4c"). InnerVolumeSpecName "kube-api-access-hjjwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:19.796708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.796677 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65201b9-871c-4418-8b36-3e6fb000cc4c-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:19.796708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.796703 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjjwd\" (UniqueName: \"kubernetes.io/projected/f65201b9-871c-4418-8b36-3e6fb000cc4c-kube-api-access-hjjwd\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:19.953762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:19.953725 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:20.185621 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.185581 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:20.568406 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.568378 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" Sep 16 13:30:20.568847 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.568377 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w" event={"ID":"f65201b9-871c-4418-8b36-3e6fb000cc4c","Type":"ContainerDied","Data":"81b65e5b229c78dae5f8e351cfccbef531fce0190acc7703082364651cc31ad3"} Sep 16 13:30:20.568847 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.568512 2581 scope.go:117] "RemoveContainer" containerID="0436ca2e86eff6b9be7ae89bf6fe35da7fc0ab87c162b6c070e4a7358f0a3606" Sep 16 13:30:20.590354 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.590331 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w"] Sep 16 13:30:20.593079 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.593060 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b97e79df-5657d8c98b-xkl7w"] Sep 16 13:30:20.870932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:20.870835 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:21.386910 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:21.386876 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fcca3b2-8e48-48dd-964d-69fbc9b91cea" path="/var/lib/kubelet/pods/4fcca3b2-8e48-48dd-964d-69fbc9b91cea/volumes" Sep 16 13:30:21.387393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:21.387371 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65201b9-871c-4418-8b36-3e6fb000cc4c" path="/var/lib/kubelet/pods/f65201b9-871c-4418-8b36-3e6fb000cc4c/volumes" Sep 16 13:30:21.655127 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:21.655046 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:22.472969 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:22.472911 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:25.696073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:25.696035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:25.816790 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:25.816760 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:25.986599 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:25.986566 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd"] Sep 16 13:30:25.986835 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:25.986802 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" containerID="cri-o://aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88" gracePeriod=30 Sep 16 13:30:26.034178 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:26.034146 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:26.342376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:26.342247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:26.430947 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:26.430918 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:26.687265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:26.687166 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:31.243630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.243605 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:30:31.300525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300457 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b558009-2356-43e2-bcb1-c13e7579966b-istio-podinfo\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300506 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-credential-socket\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300537 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-certs\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300555 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b558009-2356-43e2-bcb1-c13e7579966b-istiod-ca-cert\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300592 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76wmx\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-kube-api-access-76wmx\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300614 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-data\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300655 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-socket\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300683 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-istio-token\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.300735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300728 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-envoy\") pod \"7b558009-2356-43e2-bcb1-c13e7579966b\" (UID: \"7b558009-2356-43e2-bcb1-c13e7579966b\") " Sep 16 13:30:31.301072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300868 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:30:31.301072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300889 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:30:31.301072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.300929 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:30:31.301072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301013 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:30:31.301072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301044 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b558009-2356-43e2-bcb1-c13e7579966b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:30:31.301336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301120 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.301336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301136 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b558009-2356-43e2-bcb1-c13e7579966b-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.301336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301147 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.301336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301154 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.301336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.301162 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.303073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.303049 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b558009-2356-43e2-bcb1-c13e7579966b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:30:31.303467 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.303433 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-kube-api-access-76wmx" (OuterVolumeSpecName: "kube-api-access-76wmx") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "kube-api-access-76wmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:31.303467 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.303453 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:30:31.303671 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.303509 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b558009-2356-43e2-bcb1-c13e7579966b" (UID: "7b558009-2356-43e2-bcb1-c13e7579966b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:30:31.402255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.402224 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wmx\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-kube-api-access-76wmx\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.402255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.402246 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b558009-2356-43e2-bcb1-c13e7579966b-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.402255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.402255 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b558009-2356-43e2-bcb1-c13e7579966b-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.402451 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.402262 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b558009-2356-43e2-bcb1-c13e7579966b-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:30:31.636589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.636499 2581 generic.go:356] "Generic (PLEG): container finished" podID="7b558009-2356-43e2-bcb1-c13e7579966b" containerID="aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88" exitCode=0 Sep 16 13:30:31.636589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.636546 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" Sep 16 13:30:31.636772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.636586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" event={"ID":"7b558009-2356-43e2-bcb1-c13e7579966b","Type":"ContainerDied","Data":"aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88"} Sep 16 13:30:31.636772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.636646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd" event={"ID":"7b558009-2356-43e2-bcb1-c13e7579966b","Type":"ContainerDied","Data":"624970c60dc1e7aec1898a6202e9c67d0c5adf4e3ccf7069dd610e25bc5e9971"} Sep 16 13:30:31.636772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.636676 2581 scope.go:117] "RemoveContainer" containerID="aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88" Sep 16 13:30:31.646262 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.646237 2581 scope.go:117] "RemoveContainer" containerID="905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61" Sep 16 13:30:31.653646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.653621 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd"] Sep 16 13:30:31.655516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.655497 2581 scope.go:117] "RemoveContainer" containerID="aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88" Sep 16 13:30:31.655809 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:30:31.655789 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88\": container with ID starting with aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88 not found: ID does not exist" containerID="aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88" Sep 16 13:30:31.655865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.655817 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88"} err="failed to get container status \"aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88\": rpc error: code = NotFound desc = could not find container \"aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88\": container with ID starting with aa1cb11efd90cd81bd672635ddc5e270cf50eecd9a1a4627f6f26a442ff15c88 not found: ID does not exist" Sep 16 13:30:31.655865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.655837 2581 scope.go:117] "RemoveContainer" containerID="905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61" Sep 16 13:30:31.656135 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:30:31.656112 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61\": container with ID starting with 905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61 not found: ID does not exist" containerID="905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61" Sep 16 13:30:31.656190 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.656143 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61"} err="failed to get container status \"905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61\": rpc error: code = NotFound desc = could not find container \"905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61\": container with ID starting with 905c26a9b943708353b3b0024049659942ee89ce3cb715fdfb0a432f2327bc61 not found: ID does not exist" Sep 16 13:30:31.657793 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:31.657774 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b97e79df-openshift-default-674c78cfc6-jvjcd"] Sep 16 13:30:32.278006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:32.277968 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:32.378716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:32.378685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:32.477769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:32.477735 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:32.659779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:32.659688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:33.387181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:33.387144 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" path="/var/lib/kubelet/pods/7b558009-2356-43e2-bcb1-c13e7579966b/volumes" Sep 16 13:30:35.563365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:35.563335 2581 scope.go:117] "RemoveContainer" containerID="d862b5680798cd5d06b7e05501844f8c9a4c2cac3e433e4f4e384686248591a2" Sep 16 13:30:35.574235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:35.574216 2581 scope.go:117] "RemoveContainer" containerID="cf56b139918243a215ce565ce864f61a71c1ef22ef5ded512d55bdfdbcbe897d" Sep 16 13:30:38.595506 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:38.595467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:38.658988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:38.658955 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:39.494273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.494228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:39.819711 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.819627 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd"] Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820026 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65201b9-871c-4418-8b36-3e6fb000cc4c" containerName="payload-pre-processing" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820037 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65201b9-871c-4418-8b36-3e6fb000cc4c" containerName="payload-pre-processing" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820061 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820066 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820076 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820082 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820092 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcca3b2-8e48-48dd-964d-69fbc9b91cea" containerName="revoke-keys" Sep 16 13:30:39.820138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820097 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcca3b2-8e48-48dd-964d-69fbc9b91cea" containerName="revoke-keys" Sep 16 13:30:39.820593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820164 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" Sep 16 13:30:39.820593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820172 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcca3b2-8e48-48dd-964d-69fbc9b91cea" containerName="revoke-keys" Sep 16 13:30:39.820593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820180 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65201b9-871c-4418-8b36-3e6fb000cc4c" containerName="payload-pre-processing" Sep 16 13:30:39.820593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.820185 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b558009-2356-43e2-bcb1-c13e7579966b" containerName="istio-proxy" Sep 16 13:30:39.824697 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.824680 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.827709 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.827684 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b0514e-openshift-default-dockercfg-z7887\"" Sep 16 13:30:39.839030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.839008 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd"] Sep 16 13:30:39.865937 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.865911 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:39.879011 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.878986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-socket\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879161 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-credential-socket\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-certs\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-data\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879334 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-token\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879371 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrg2\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-kube-api-access-btrg2\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879406 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-envoy\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.879449 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.879422 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-podinfo\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-podinfo\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980753 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980590 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-socket\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980753 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-credential-socket\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-certs\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980973 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-data\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980973 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-token\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.980973 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrg2\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-kube-api-access-btrg2\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.981134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.980988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-envoy\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.981192 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.981149 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-credential-socket\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.981401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.981370 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-socket\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.981555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.981431 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-certs\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.981555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.981461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.981687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.981552 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-data\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.983329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.983275 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-podinfo\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.983717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.983690 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-envoy\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.997516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.997494 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-token\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:39.997914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:39.997894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrg2\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-kube-api-access-btrg2\") pod \"e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:40.134710 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.134641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:40.155888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.155856 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:40.415647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.415621 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd"] Sep 16 13:30:40.416643 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:30:40.416620 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada1219c_edb8_4cce_81b0_f30a866f9ed1.slice/crio-27b4664f06be1c9bcc875f6951f401611c3edbcb27a4381f86298f5fb9529347 WatchSource:0}: Error finding container 27b4664f06be1c9bcc875f6951f401611c3edbcb27a4381f86298f5fb9529347: Status 404 returned error can't find the container with id 27b4664f06be1c9bcc875f6951f401611c3edbcb27a4381f86298f5fb9529347 Sep 16 13:30:40.418881 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.418849 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:30:40.418967 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.418911 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:30:40.418967 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.418939 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:30:40.692436 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.692350 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" event={"ID":"ada1219c-edb8-4cce-81b0-f30a866f9ed1","Type":"ContainerStarted","Data":"35fddaeda145991bc8b42607e1b0bb6bfba6a3f71e9f9cd2dcfac817ef697bc3"} Sep 16 13:30:40.692436 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.692413 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" event={"ID":"ada1219c-edb8-4cce-81b0-f30a866f9ed1","Type":"ContainerStarted","Data":"27b4664f06be1c9bcc875f6951f401611c3edbcb27a4381f86298f5fb9529347"} Sep 16 13:30:40.718522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:40.718469 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podStartSLOduration=1.718452538 podStartE2EDuration="1.718452538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:40.717650952 +0000 UTC m=+1325.837377537" watchObservedRunningTime="2026-09-16 13:30:40.718452538 +0000 UTC m=+1325.838179086" Sep 16 13:30:41.135850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:41.135817 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:41.140641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:41.140617 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:41.698566 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:41.698530 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:41.699571 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:41.699545 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:30:42.604756 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:42.604714 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:42.704040 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:42.703997 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:42.805818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:42.805772 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:43.451115 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:43.451077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:44.478122 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:44.478091 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:45.372393 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:45.372354 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:45.475222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:45.475180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:45.796453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:45.796422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:45.900822 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:45.900786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:45.998959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:45.998926 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz"] Sep 16 13:30:46.000376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.000336 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:46.002966 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.002950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.005462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.005441 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b0514e-dockercfg-xjk9m\"" Sep 16 13:30:46.005587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.005489 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b0514e\"" Sep 16 13:30:46.020673 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.020652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz"] Sep 16 13:30:46.142568 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.142484 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.142712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.142567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52fm6\" (UniqueName: \"kubernetes.io/projected/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-kube-api-access-52fm6\") pod \"payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.243932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.243901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52fm6\" (UniqueName: \"kubernetes.io/projected/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-kube-api-access-52fm6\") pod \"payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.244088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.243984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.244682 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.244663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.253001 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.252976 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fm6\" (UniqueName: \"kubernetes.io/projected/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-kube-api-access-52fm6\") pod \"payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.313806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.313776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.590566 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.590539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz"] Sep 16 13:30:46.592401 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:30:46.592375 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9982d77_4da2_4a7b_8af8_a2bbfb8eab23.slice/crio-a9cb08430046649151e3244f01e02acda719ae9489ddac4bf6c18803dd93d0b6 WatchSource:0}: Error finding container a9cb08430046649151e3244f01e02acda719ae9489ddac4bf6c18803dd93d0b6: Status 404 returned error can't find the container with id a9cb08430046649151e3244f01e02acda719ae9489ddac4bf6c18803dd93d0b6 Sep 16 13:30:46.731459 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.731420 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" event={"ID":"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23","Type":"ContainerStarted","Data":"cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8"} Sep 16 13:30:46.731641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.731491 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" event={"ID":"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23","Type":"ContainerStarted","Data":"a9cb08430046649151e3244f01e02acda719ae9489ddac4bf6c18803dd93d0b6"} Sep 16 13:30:46.731641 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.731522 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:30:46.754513 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:46.754466 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" podStartSLOduration=1.754453099 podStartE2EDuration="1.754453099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:46.751799159 +0000 UTC m=+1331.871525733" watchObservedRunningTime="2026-09-16 13:30:46.754453099 +0000 UTC m=+1331.874179640" Sep 16 13:30:49.761417 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:49.761384 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:56.039664 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:56.039626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:30:57.741817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:30:57.741780 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:31:00.833168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:00.833088 2581 generic.go:356] "Generic (PLEG): container finished" podID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerID="35fddaeda145991bc8b42607e1b0bb6bfba6a3f71e9f9cd2dcfac817ef697bc3" exitCode=0 Sep 16 13:31:00.833168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:00.833143 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" event={"ID":"ada1219c-edb8-4cce-81b0-f30a866f9ed1","Type":"ContainerDied","Data":"35fddaeda145991bc8b42607e1b0bb6bfba6a3f71e9f9cd2dcfac817ef697bc3"} Sep 16 13:31:00.833662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:00.833538 2581 scope.go:117] "RemoveContainer" containerID="35fddaeda145991bc8b42607e1b0bb6bfba6a3f71e9f9cd2dcfac817ef697bc3" Sep 16 13:31:00.834636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:00.834605 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:31:00.834708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:00.834674 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:31:00.834745 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:00.834706 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:31:01.841598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:01.841566 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" event={"ID":"ada1219c-edb8-4cce-81b0-f30a866f9ed1","Type":"ContainerStarted","Data":"658871916f2f1e3b0e967f0297ca29eccf00d257336e411417659c828cfb8ab0"} Sep 16 13:31:02.135589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:02.135494 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:31:02.135747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:02.135727 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 16 13:31:02.135811 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:02.135790 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 16 13:31:03.135859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:03.135820 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 16 13:31:03.136302 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:03.135879 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 16 13:31:04.135257 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:04.135211 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 16 13:31:04.135470 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:04.135274 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 16 13:31:05.135841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:05.135802 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 16 13:31:05.136339 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:05.135864 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 16 13:31:06.135731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:06.135697 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 16 13:31:06.135912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:06.135754 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 16 13:31:07.180259 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:07.180224 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:31:07.180731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:07.180633 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:31:07.181472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:07.181454 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:31:07.528323 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:07.528263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:07.674010 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:07.673977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:08.050215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:08.050180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:08.409345 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:08.409258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:08.517528 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:08.517491 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:08.620589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:08.620542 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:09.262950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:09.262916 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:10.145116 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:10.145074 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:11.026365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:11.026326 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:11.630352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:11.630318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:11.727314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:11.727270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:12.098008 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:12.097973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:12.958872 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:12.958841 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:13.927381 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:13.927271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:14.715452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:14.715416 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:15.455919 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:15.455882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:16.231015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.230977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:16.328137 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.328101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:16.864961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.864930 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74"] Sep 16 13:31:16.868655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.868637 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:16.871330 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.871308 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7841d9-dockercfg-svxrk\"" Sep 16 13:31:16.871431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.871359 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7841d9\"" Sep 16 13:31:16.935222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.924695 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74"] Sep 16 13:31:16.961356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.961249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:16.961356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:16.961345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6fh\" (UniqueName: \"kubernetes.io/projected/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-kube-api-access-2b6fh\") pod \"payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.062138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.062099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.062138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.062144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6fh\" (UniqueName: \"kubernetes.io/projected/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-kube-api-access-2b6fh\") pod \"payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.062825 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.062802 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.064417 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.064386 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:17.071015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.070990 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6fh\" (UniqueName: \"kubernetes.io/projected/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-kube-api-access-2b6fh\") pod \"payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.161591 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.161488 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:17.178404 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.178381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.460871 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.460837 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74"] Sep 16 13:31:17.463010 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:31:17.462981 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe13d5bd_0ea7_4e26_a122_e173d0a2f78c.slice/crio-9e3d068b28302e8780172d9764ccf24a03fd71563fefe83a7ffc115446435c10 WatchSource:0}: Error finding container 9e3d068b28302e8780172d9764ccf24a03fd71563fefe83a7ffc115446435c10: Status 404 returned error can't find the container with id 9e3d068b28302e8780172d9764ccf24a03fd71563fefe83a7ffc115446435c10 Sep 16 13:31:17.950690 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.950656 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" event={"ID":"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c","Type":"ContainerStarted","Data":"f83f315ecfdd55e652aa4794f2a7e4d6c5dc656ee70d056bd2c9cf1ef562c919"} Sep 16 13:31:17.950884 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.950713 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" event={"ID":"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c","Type":"ContainerStarted","Data":"9e3d068b28302e8780172d9764ccf24a03fd71563fefe83a7ffc115446435c10"} Sep 16 13:31:17.950884 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.950790 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:17.959310 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.959255 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:17.968609 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:17.968570 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" podStartSLOduration=1.968555999 podStartE2EDuration="1.968555999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:17.967176279 +0000 UTC m=+1363.086902827" watchObservedRunningTime="2026-09-16 13:31:17.968555999 +0000 UTC m=+1363.088282605" Sep 16 13:31:28.649789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:28.649751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:28.960177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:28.960147 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:31:32.011906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:32.011869 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:37.133925 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:37.133890 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:38.614094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:38.614053 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:39.455716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:39.455677 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:40.351199 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:40.351166 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:41.187031 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:41.186996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:41.610906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:41.610873 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:41.978661 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:41.978621 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:42.937868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:42.937831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:50.279589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:50.279549 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:51.302672 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.302644 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:51.702944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.702915 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm"] Sep 16 13:31:51.706877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.706859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.709689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.709661 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a235ace-openshift-default-dockercfg-kxnng\"" Sep 16 13:31:51.729438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.729411 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm"] Sep 16 13:31:51.744156 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.744133 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.792792 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-podinfo\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.792866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-data\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.792901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-token\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.792938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-certs\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.792975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-credential-socket\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.793032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcfx\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-kube-api-access-8mcfx\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.793114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-socket\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.793140 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efc3a8d-72a1-4177-991f-e24f53025a69-istiod-ca-cert\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.794301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.793175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-envoy\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.893740 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-data\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.893932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-token\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.893932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-certs\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.893932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-credential-socket\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.893932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcfx\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-kube-api-access-8mcfx\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-socket\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efc3a8d-72a1-4177-991f-e24f53025a69-istiod-ca-cert\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.893991 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-envoy\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.894062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-podinfo\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.894143 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-data\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.894194 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-credential-socket\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.894339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-socket\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894587 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.894563 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-certs\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.894858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.894837 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efc3a8d-72a1-4177-991f-e24f53025a69-istiod-ca-cert\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.896930 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.896908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-envoy\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.897115 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.897096 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-podinfo\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.902314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.902272 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-token\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:51.902314 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:51.902294 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcfx\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-kube-api-access-8mcfx\") pod \"e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:52.018804 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:52.018708 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:52.119465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:52.119423 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:52.309963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:52.308259 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm"] Sep 16 13:31:52.310963 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:31:52.310931 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efc3a8d_72a1_4177_991f_e24f53025a69.slice/crio-da0857db8f14df0bfb1d5699eaa2152344161d9dbcf323aaaf8b13f4d108b0e8 WatchSource:0}: Error finding container da0857db8f14df0bfb1d5699eaa2152344161d9dbcf323aaaf8b13f4d108b0e8: Status 404 returned error can't find the container with id da0857db8f14df0bfb1d5699eaa2152344161d9dbcf323aaaf8b13f4d108b0e8 Sep 16 13:31:52.313892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:52.313852 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:31:52.313976 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:52.313938 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:31:52.314016 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:52.313980 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:31:53.165479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:53.165439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" event={"ID":"9efc3a8d-72a1-4177-991f-e24f53025a69","Type":"ContainerStarted","Data":"7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da"} Sep 16 13:31:53.165676 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:53.165508 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" event={"ID":"9efc3a8d-72a1-4177-991f-e24f53025a69","Type":"ContainerStarted","Data":"da0857db8f14df0bfb1d5699eaa2152344161d9dbcf323aaaf8b13f4d108b0e8"} Sep 16 13:31:53.188363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:53.188313 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podStartSLOduration=2.188296818 podStartE2EDuration="2.188296818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:53.185725899 +0000 UTC m=+1398.305452468" watchObservedRunningTime="2026-09-16 13:31:53.188296818 +0000 UTC m=+1398.308023360" Sep 16 13:31:53.218458 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:53.218428 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:53.278102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:53.278070 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:54.018877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:54.018838 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:54.024140 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:54.024117 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:54.173154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:54.173123 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:54.174137 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:54.174114 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:31:54.473453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:54.473421 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:55.142011 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:55.141970 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:55.235762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:55.235733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:55.409749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:55.409669 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:56.372933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:56.372886 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:57.245170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:57.245138 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:58.157372 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:58.157331 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:58.943374 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:58.943335 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:59.726226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:59.726185 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:59.820783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:31:59.820747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:00.400529 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:00.400501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:00.503225 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:00.503187 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:19.990861 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:19.990810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.227850 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.227813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.347932 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.347900 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx"] Sep 16 13:32:20.351679 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.351662 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.354296 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.354263 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:32:20.355179 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.355160 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:32:20.355243 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.355178 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:32:20.361718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.361696 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx"] Sep 16 13:32:20.466341 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.466303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pqd\" (UniqueName: \"kubernetes.io/projected/3fea35b3-1421-4dd8-b67b-407f4aaa315e-kube-api-access-r8pqd\") pod \"maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.466530 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.466424 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fea35b3-1421-4dd8-b67b-407f4aaa315e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.567315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.567188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fea35b3-1421-4dd8-b67b-407f4aaa315e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.567506 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.567343 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pqd\" (UniqueName: \"kubernetes.io/projected/3fea35b3-1421-4dd8-b67b-407f4aaa315e-kube-api-access-r8pqd\") pod \"maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.568085 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.568062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fea35b3-1421-4dd8-b67b-407f4aaa315e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.578750 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.578717 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pqd\" (UniqueName: \"kubernetes.io/projected/3fea35b3-1421-4dd8-b67b-407f4aaa315e-kube-api-access-r8pqd\") pod \"maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.626276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.626247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:20.663956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.663929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:20.966651 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:20.966619 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx"] Sep 16 13:32:20.968262 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:32:20.968227 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fea35b3_1421_4dd8_b67b_407f4aaa315e.slice/crio-f423be996e2355aa6249011e2a08f5379f282f71a4fbccebef760ff7acc217ce WatchSource:0}: Error finding container f423be996e2355aa6249011e2a08f5379f282f71a4fbccebef760ff7acc217ce: Status 404 returned error can't find the container with id f423be996e2355aa6249011e2a08f5379f282f71a4fbccebef760ff7acc217ce Sep 16 13:32:21.349817 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:21.349783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" event={"ID":"3fea35b3-1421-4dd8-b67b-407f4aaa315e","Type":"ContainerStarted","Data":"a77e1d4725ca18681f19f7775f0d6aec8dca1b3c2e881dc4ff2377a3e82d9570"} Sep 16 13:32:21.350185 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:21.349845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" event={"ID":"3fea35b3-1421-4dd8-b67b-407f4aaa315e","Type":"ContainerStarted","Data":"f423be996e2355aa6249011e2a08f5379f282f71a4fbccebef760ff7acc217ce"} Sep 16 13:32:21.378354 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:21.378309 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" podStartSLOduration=1.378294324 podStartE2EDuration="1.378294324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:21.374462238 +0000 UTC m=+1426.494188792" watchObservedRunningTime="2026-09-16 13:32:21.378294324 +0000 UTC m=+1426.498020863" Sep 16 13:32:21.864335 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:21.864251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:22.357731 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:22.357688 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fea35b3-1421-4dd8-b67b-407f4aaa315e" containerID="a77e1d4725ca18681f19f7775f0d6aec8dca1b3c2e881dc4ff2377a3e82d9570" exitCode=0 Sep 16 13:32:22.358151 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:22.357767 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" event={"ID":"3fea35b3-1421-4dd8-b67b-407f4aaa315e","Type":"ContainerDied","Data":"a77e1d4725ca18681f19f7775f0d6aec8dca1b3c2e881dc4ff2377a3e82d9570"} Sep 16 13:32:22.847058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:22.847014 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:23.365462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.365430 2581 generic.go:356] "Generic (PLEG): container finished" podID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerID="7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da" exitCode=0 Sep 16 13:32:23.365927 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.365497 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" event={"ID":"9efc3a8d-72a1-4177-991f-e24f53025a69","Type":"ContainerDied","Data":"7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da"} Sep 16 13:32:23.366212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.365970 2581 scope.go:117] "RemoveContainer" containerID="7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da" Sep 16 13:32:23.367165 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.367131 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:32:23.367308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.367210 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:32:23.367308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.367242 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 13:32:23.538241 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.538215 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:23.597340 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.597310 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fea35b3-1421-4dd8-b67b-407f4aaa315e-maas-api-service-ca\") pod \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " Sep 16 13:32:23.597526 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.597383 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8pqd\" (UniqueName: \"kubernetes.io/projected/3fea35b3-1421-4dd8-b67b-407f4aaa315e-kube-api-access-r8pqd\") pod \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\" (UID: \"3fea35b3-1421-4dd8-b67b-407f4aaa315e\") " Sep 16 13:32:23.597919 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.597895 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fea35b3-1421-4dd8-b67b-407f4aaa315e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fea35b3-1421-4dd8-b67b-407f4aaa315e" (UID: "3fea35b3-1421-4dd8-b67b-407f4aaa315e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:32:23.599554 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.599532 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fea35b3-1421-4dd8-b67b-407f4aaa315e-kube-api-access-r8pqd" (OuterVolumeSpecName: "kube-api-access-r8pqd") pod "3fea35b3-1421-4dd8-b67b-407f4aaa315e" (UID: "3fea35b3-1421-4dd8-b67b-407f4aaa315e"). InnerVolumeSpecName "kube-api-access-r8pqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:32:23.643488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.643398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:23.701856 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.698733 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pqd\" (UniqueName: \"kubernetes.io/projected/3fea35b3-1421-4dd8-b67b-407f4aaa315e-kube-api-access-r8pqd\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:32:23.701856 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:23.698765 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fea35b3-1421-4dd8-b67b-407f4aaa315e-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:32:24.173905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:24.173874 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:32:24.373599 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:24.373561 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" event={"ID":"9efc3a8d-72a1-4177-991f-e24f53025a69","Type":"ContainerStarted","Data":"8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0"} Sep 16 13:32:24.374978 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:24.374954 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" event={"ID":"3fea35b3-1421-4dd8-b67b-407f4aaa315e","Type":"ContainerDied","Data":"f423be996e2355aa6249011e2a08f5379f282f71a4fbccebef760ff7acc217ce"} Sep 16 13:32:24.374978 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:24.374976 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx" Sep 16 13:32:24.375116 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:24.374998 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f423be996e2355aa6249011e2a08f5379f282f71a4fbccebef760ff7acc217ce" Sep 16 13:32:24.455762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:24.455733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:25.019113 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:25.019070 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:32:25.020156 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:25.020129 2581 patch_prober.go:28] interesting pod/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 16 13:32:25.020313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:25.020180 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 16 13:32:25.463586 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:25.463552 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:26.019489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:26.019454 2581 patch_prober.go:28] interesting pod/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 16 13:32:26.019681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:26.019509 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 16 13:32:26.480785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:26.480755 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:27.019410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:27.019375 2581 patch_prober.go:28] interesting pod/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 16 13:32:27.019609 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:27.019432 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 16 13:32:27.414438 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:27.414348 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:28.019715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:28.019628 2581 patch_prober.go:28] interesting pod/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 16 13:32:28.019715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:28.019701 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 16 13:32:28.120304 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:28.120239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:29.019965 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:29.019930 2581 patch_prober.go:28] interesting pod/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 16 13:32:29.020426 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:29.019995 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 16 13:32:29.087981 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:29.087949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:29.818047 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:29.818016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:30.023922 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:30.023887 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:32:30.024405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:30.024265 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:32:30.025021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:30.025003 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:32:30.376268 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:30.376235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:30.679577 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:30.679481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:31.746949 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:31.746916 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:35.616171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:35.616139 2581 scope.go:117] "RemoveContainer" containerID="0399822b65c674c28f8ef0aaad38bd9f02909403f379b957c4e5b20edc2d7bf9" Sep 16 13:32:42.296073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:42.296031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:45.274430 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:45.274390 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:32:53.044319 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:53.044271 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f"] Sep 16 13:32:53.048250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:53.048217 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-48adf5-xt88f"] Sep 16 13:32:53.387828 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:32:53.387747 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c2ec955-7ae2-4011-ab47-d15e542da569" path="/var/lib/kubelet/pods/3c2ec955-7ae2-4011-ab47-d15e542da569/volumes" Sep 16 13:33:31.341727 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:33:31.341679 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:35.473491 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:33:35.473465 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:33:35.477057 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:33:35.477035 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:33:51.329152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:33:51.329121 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:51.430177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:33:51.430149 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:33:57.414783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:33:57.414747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:00.548250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:00.548172 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:35.653049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:35.653018 2581 scope.go:117] "RemoveContainer" containerID="82603e24b45ef72efd4bb72987a2e51fff53b8e5d2ed6a9b78dec764939585e4" Sep 16 13:34:40.576106 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.576075 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:40.701805 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.701776 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624"] Sep 16 13:34:40.702228 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.702215 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fea35b3-1421-4dd8-b67b-407f4aaa315e" containerName="revoke-keys" Sep 16 13:34:40.702272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.702231 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fea35b3-1421-4dd8-b67b-407f4aaa315e" containerName="revoke-keys" Sep 16 13:34:40.702363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.702353 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fea35b3-1421-4dd8-b67b-407f4aaa315e" containerName="revoke-keys" Sep 16 13:34:40.705477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.705461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:40.707948 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.707928 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:34:40.708952 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.708920 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:34:40.709058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.708951 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:34:40.729356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.729334 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624"] Sep 16 13:34:40.790610 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.790581 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df13f358-1ef6-4b3d-8094-8686fb83e8b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:40.790735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.790677 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdn96\" (UniqueName: \"kubernetes.io/projected/df13f358-1ef6-4b3d-8094-8686fb83e8b3-kube-api-access-tdn96\") pod \"maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:40.891792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.891716 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df13f358-1ef6-4b3d-8094-8686fb83e8b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:40.891931 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.891800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdn96\" (UniqueName: \"kubernetes.io/projected/df13f358-1ef6-4b3d-8094-8686fb83e8b3-kube-api-access-tdn96\") pod \"maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:40.892412 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.892394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df13f358-1ef6-4b3d-8094-8686fb83e8b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:40.901670 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:40.901650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdn96\" (UniqueName: \"kubernetes.io/projected/df13f358-1ef6-4b3d-8094-8686fb83e8b3-kube-api-access-tdn96\") pod \"maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:41.016205 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:41.016171 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:41.293700 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:41.293677 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624"] Sep 16 13:34:41.296133 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:34:41.296106 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf13f358_1ef6_4b3d_8094_8686fb83e8b3.slice/crio-07c78b4c1990e9468a280ccbf097b7389b6d936dd556438d8e222a319aa27b83 WatchSource:0}: Error finding container 07c78b4c1990e9468a280ccbf097b7389b6d936dd556438d8e222a319aa27b83: Status 404 returned error can't find the container with id 07c78b4c1990e9468a280ccbf097b7389b6d936dd556438d8e222a319aa27b83 Sep 16 13:34:42.244390 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:42.244355 2581 generic.go:356] "Generic (PLEG): container finished" podID="df13f358-1ef6-4b3d-8094-8686fb83e8b3" containerID="cf562767cc8abfeb07952e3df8b4b63688ae6c62e4d0724284a6374cf53f80f4" exitCode=0 Sep 16 13:34:42.244783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:42.244418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" event={"ID":"df13f358-1ef6-4b3d-8094-8686fb83e8b3","Type":"ContainerDied","Data":"cf562767cc8abfeb07952e3df8b4b63688ae6c62e4d0724284a6374cf53f80f4"} Sep 16 13:34:42.244783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:42.244472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" event={"ID":"df13f358-1ef6-4b3d-8094-8686fb83e8b3","Type":"ContainerStarted","Data":"07c78b4c1990e9468a280ccbf097b7389b6d936dd556438d8e222a319aa27b83"} Sep 16 13:34:43.385435 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.385409 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:43.414691 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.414665 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df13f358-1ef6-4b3d-8094-8686fb83e8b3-maas-api-service-ca\") pod \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " Sep 16 13:34:43.414820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.414731 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdn96\" (UniqueName: \"kubernetes.io/projected/df13f358-1ef6-4b3d-8094-8686fb83e8b3-kube-api-access-tdn96\") pod \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\" (UID: \"df13f358-1ef6-4b3d-8094-8686fb83e8b3\") " Sep 16 13:34:43.415025 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.415001 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df13f358-1ef6-4b3d-8094-8686fb83e8b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df13f358-1ef6-4b3d-8094-8686fb83e8b3" (UID: "df13f358-1ef6-4b3d-8094-8686fb83e8b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:34:43.415130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.415117 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df13f358-1ef6-4b3d-8094-8686fb83e8b3-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:34:43.416935 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.416905 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df13f358-1ef6-4b3d-8094-8686fb83e8b3-kube-api-access-tdn96" (OuterVolumeSpecName: "kube-api-access-tdn96") pod "df13f358-1ef6-4b3d-8094-8686fb83e8b3" (UID: "df13f358-1ef6-4b3d-8094-8686fb83e8b3"). InnerVolumeSpecName "kube-api-access-tdn96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:34:43.516140 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:43.516074 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdn96\" (UniqueName: \"kubernetes.io/projected/df13f358-1ef6-4b3d-8094-8686fb83e8b3-kube-api-access-tdn96\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:34:44.257779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:44.257745 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" Sep 16 13:34:44.257956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:44.257775 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624" event={"ID":"df13f358-1ef6-4b3d-8094-8686fb83e8b3","Type":"ContainerDied","Data":"07c78b4c1990e9468a280ccbf097b7389b6d936dd556438d8e222a319aa27b83"} Sep 16 13:34:44.257956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:44.257828 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c78b4c1990e9468a280ccbf097b7389b6d936dd556438d8e222a319aa27b83" Sep 16 13:34:44.375269 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:44.375237 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:44.511751 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:44.511657 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:45.518099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:45.518067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:45.719776 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:45.719740 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:45.978736 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:45.978708 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:47.119042 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:47.119009 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:34:56.566292 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:34:56.566259 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:21.020530 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:21.020492 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:21.806196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:21.806162 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:21.906844 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:21.906807 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:23.361292 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:23.361243 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:26.891197 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:26.891141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:29.041072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:29.040996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:35:29.147887 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:35:29.147858 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:07.050923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:07.050879 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:18.089456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.089419 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg"] Sep 16 13:36:18.090084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.090062 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df13f358-1ef6-4b3d-8094-8686fb83e8b3" containerName="revoke-keys" Sep 16 13:36:18.090177 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.090087 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="df13f358-1ef6-4b3d-8094-8686fb83e8b3" containerName="revoke-keys" Sep 16 13:36:18.090243 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.090185 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="df13f358-1ef6-4b3d-8094-8686fb83e8b3" containerName="revoke-keys" Sep 16 13:36:18.093429 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.093408 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.095765 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.095744 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:36:18.096168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.096153 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:36:18.096742 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.096728 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:36:18.102961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.102937 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg"] Sep 16 13:36:18.201909 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.201875 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15883caa-b9ee-424b-a01d-a6d536e21db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a235ace-262dg\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.202109 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.201984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjf8\" (UniqueName: \"kubernetes.io/projected/15883caa-b9ee-424b-a01d-a6d536e21db1-kube-api-access-tvjf8\") pod \"maas-api-revoke-keys-e2e-mt-4a235ace-262dg\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.302631 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.302597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15883caa-b9ee-424b-a01d-a6d536e21db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a235ace-262dg\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.302823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.302683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjf8\" (UniqueName: \"kubernetes.io/projected/15883caa-b9ee-424b-a01d-a6d536e21db1-kube-api-access-tvjf8\") pod \"maas-api-revoke-keys-e2e-mt-4a235ace-262dg\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.303075 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.303047 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:18.303350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.303334 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15883caa-b9ee-424b-a01d-a6d536e21db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a235ace-262dg\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.318940 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.318916 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjf8\" (UniqueName: \"kubernetes.io/projected/15883caa-b9ee-424b-a01d-a6d536e21db1-kube-api-access-tvjf8\") pod \"maas-api-revoke-keys-e2e-mt-4a235ace-262dg\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.405052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.404960 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:18.678308 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.678261 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg"] Sep 16 13:36:18.679447 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:36:18.679417 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15883caa_b9ee_424b_a01d_a6d536e21db1.slice/crio-257b08f7f3b7d79469732e9e98e4fba741aafb153557ec92554854e2929b7e69 WatchSource:0}: Error finding container 257b08f7f3b7d79469732e9e98e4fba741aafb153557ec92554854e2929b7e69: Status 404 returned error can't find the container with id 257b08f7f3b7d79469732e9e98e4fba741aafb153557ec92554854e2929b7e69 Sep 16 13:36:18.849144 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:18.849111 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" event={"ID":"15883caa-b9ee-424b-a01d-a6d536e21db1","Type":"ContainerStarted","Data":"257b08f7f3b7d79469732e9e98e4fba741aafb153557ec92554854e2929b7e69"} Sep 16 13:36:19.303151 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.303069 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:19.423665 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.423634 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:19.529968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.529938 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z"] Sep 16 13:36:19.533477 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.533461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.544801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.544724 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z"] Sep 16 13:36:19.614336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.614234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt92v\" (UniqueName: \"kubernetes.io/projected/60986dd7-b33f-4047-93f4-88f22f7db4f5-kube-api-access-pt92v\") pod \"maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.614476 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.614377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60986dd7-b33f-4047-93f4-88f22f7db4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.715774 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.715733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt92v\" (UniqueName: \"kubernetes.io/projected/60986dd7-b33f-4047-93f4-88f22f7db4f5-kube-api-access-pt92v\") pod \"maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.715940 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.715824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60986dd7-b33f-4047-93f4-88f22f7db4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.716414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.716395 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60986dd7-b33f-4047-93f4-88f22f7db4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.725652 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.725622 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt92v\" (UniqueName: \"kubernetes.io/projected/60986dd7-b33f-4047-93f4-88f22f7db4f5-kube-api-access-pt92v\") pod \"maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.846150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.846118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:19.868544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.868510 2581 generic.go:356] "Generic (PLEG): container finished" podID="15883caa-b9ee-424b-a01d-a6d536e21db1" containerID="6f18f3e2f699483bee0221518ef1963148114764f8744c86f6c833482d502ff2" exitCode=0 Sep 16 13:36:19.868651 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:19.868597 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" event={"ID":"15883caa-b9ee-424b-a01d-a6d536e21db1","Type":"ContainerDied","Data":"6f18f3e2f699483bee0221518ef1963148114764f8744c86f6c833482d502ff2"} Sep 16 13:36:20.138694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:20.138669 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z"] Sep 16 13:36:20.139677 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:36:20.139656 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60986dd7_b33f_4047_93f4_88f22f7db4f5.slice/crio-8993dd93aa89ab721a3f11e89dca960e1d6c9bb405ed38a28d536780cdcd35ed WatchSource:0}: Error finding container 8993dd93aa89ab721a3f11e89dca960e1d6c9bb405ed38a28d536780cdcd35ed: Status 404 returned error can't find the container with id 8993dd93aa89ab721a3f11e89dca960e1d6c9bb405ed38a28d536780cdcd35ed Sep 16 13:36:20.258722 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:20.258686 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:20.876675 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:20.876640 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" event={"ID":"60986dd7-b33f-4047-93f4-88f22f7db4f5","Type":"ContainerStarted","Data":"200c904f7e191f46435859e5c7dffb6b7ddd35e36f894899520cff0c2d5a1b6b"} Sep 16 13:36:20.877103 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:20.876692 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" event={"ID":"60986dd7-b33f-4047-93f4-88f22f7db4f5","Type":"ContainerStarted","Data":"8993dd93aa89ab721a3f11e89dca960e1d6c9bb405ed38a28d536780cdcd35ed"} Sep 16 13:36:20.901455 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:20.901405 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" podStartSLOduration=1.901387164 podStartE2EDuration="1.901387164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:20.899840488 +0000 UTC m=+1666.019567060" watchObservedRunningTime="2026-09-16 13:36:20.901387164 +0000 UTC m=+1666.021113713" Sep 16 13:36:21.039884 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.039863 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:21.130020 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.129948 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvjf8\" (UniqueName: \"kubernetes.io/projected/15883caa-b9ee-424b-a01d-a6d536e21db1-kube-api-access-tvjf8\") pod \"15883caa-b9ee-424b-a01d-a6d536e21db1\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " Sep 16 13:36:21.130162 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.130089 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15883caa-b9ee-424b-a01d-a6d536e21db1-maas-api-service-ca\") pod \"15883caa-b9ee-424b-a01d-a6d536e21db1\" (UID: \"15883caa-b9ee-424b-a01d-a6d536e21db1\") " Sep 16 13:36:21.130469 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.130440 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15883caa-b9ee-424b-a01d-a6d536e21db1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15883caa-b9ee-424b-a01d-a6d536e21db1" (UID: "15883caa-b9ee-424b-a01d-a6d536e21db1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:36:21.132012 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.131988 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15883caa-b9ee-424b-a01d-a6d536e21db1-kube-api-access-tvjf8" (OuterVolumeSpecName: "kube-api-access-tvjf8") pod "15883caa-b9ee-424b-a01d-a6d536e21db1" (UID: "15883caa-b9ee-424b-a01d-a6d536e21db1"). InnerVolumeSpecName "kube-api-access-tvjf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:36:21.231759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.231725 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15883caa-b9ee-424b-a01d-a6d536e21db1-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:21.231759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.231752 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjf8\" (UniqueName: \"kubernetes.io/projected/15883caa-b9ee-424b-a01d-a6d536e21db1-kube-api-access-tvjf8\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:21.631653 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.631604 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:21.884488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.884459 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" Sep 16 13:36:21.884889 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.884506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg" event={"ID":"15883caa-b9ee-424b-a01d-a6d536e21db1","Type":"ContainerDied","Data":"257b08f7f3b7d79469732e9e98e4fba741aafb153557ec92554854e2929b7e69"} Sep 16 13:36:21.884889 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.884560 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="257b08f7f3b7d79469732e9e98e4fba741aafb153557ec92554854e2929b7e69" Sep 16 13:36:21.886311 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.886255 2581 generic.go:356] "Generic (PLEG): container finished" podID="60986dd7-b33f-4047-93f4-88f22f7db4f5" containerID="200c904f7e191f46435859e5c7dffb6b7ddd35e36f894899520cff0c2d5a1b6b" exitCode=0 Sep 16 13:36:21.886464 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:21.886324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" event={"ID":"60986dd7-b33f-4047-93f4-88f22f7db4f5","Type":"ContainerDied","Data":"200c904f7e191f46435859e5c7dffb6b7ddd35e36f894899520cff0c2d5a1b6b"} Sep 16 13:36:23.031487 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.031460 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:23.150723 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.150621 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt92v\" (UniqueName: \"kubernetes.io/projected/60986dd7-b33f-4047-93f4-88f22f7db4f5-kube-api-access-pt92v\") pod \"60986dd7-b33f-4047-93f4-88f22f7db4f5\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " Sep 16 13:36:23.150963 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.150769 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60986dd7-b33f-4047-93f4-88f22f7db4f5-maas-api-service-ca\") pod \"60986dd7-b33f-4047-93f4-88f22f7db4f5\" (UID: \"60986dd7-b33f-4047-93f4-88f22f7db4f5\") " Sep 16 13:36:23.151167 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.151142 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60986dd7-b33f-4047-93f4-88f22f7db4f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60986dd7-b33f-4047-93f4-88f22f7db4f5" (UID: "60986dd7-b33f-4047-93f4-88f22f7db4f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:36:23.152685 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.152664 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60986dd7-b33f-4047-93f4-88f22f7db4f5-kube-api-access-pt92v" (OuterVolumeSpecName: "kube-api-access-pt92v") pod "60986dd7-b33f-4047-93f4-88f22f7db4f5" (UID: "60986dd7-b33f-4047-93f4-88f22f7db4f5"). InnerVolumeSpecName "kube-api-access-pt92v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:36:23.183395 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.183360 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:23.252231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.252191 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60986dd7-b33f-4047-93f4-88f22f7db4f5-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:23.252231 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.252226 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt92v\" (UniqueName: \"kubernetes.io/projected/60986dd7-b33f-4047-93f4-88f22f7db4f5-kube-api-access-pt92v\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:23.900612 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.900585 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" Sep 16 13:36:23.900886 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.900629 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z" event={"ID":"60986dd7-b33f-4047-93f4-88f22f7db4f5","Type":"ContainerDied","Data":"8993dd93aa89ab721a3f11e89dca960e1d6c9bb405ed38a28d536780cdcd35ed"} Sep 16 13:36:23.900886 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:23.900685 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8993dd93aa89ab721a3f11e89dca960e1d6c9bb405ed38a28d536780cdcd35ed" Sep 16 13:36:24.451226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:24.451198 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv"] Sep 16 13:36:24.727603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:24.727527 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:24.903373 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:24.903337 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:26.153607 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:26.153564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:26.602250 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:26.602219 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:26.808619 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:26.808574 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:28.332164 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:28.332121 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:29.717493 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:29.717461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:35.686545 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:35.686514 2581 scope.go:117] "RemoveContainer" containerID="94efe9d177d0f396157f34b5288ed94cbbee842ac38406fc88b906b2973744ec" Sep 16 13:36:36.125759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:36.125722 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:36.300796 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:36.300763 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:36.506728 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:36.506695 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:36.684138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:36.684105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:36.724171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:36.724142 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg"] Sep 16 13:36:36.735411 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:36.735387 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a235ace-262dg"] Sep 16 13:36:37.120812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:37.120779 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:37.387837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:37.387758 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15883caa-b9ee-424b-a01d-a6d536e21db1" path="/var/lib/kubelet/pods/15883caa-b9ee-424b-a01d-a6d536e21db1/volumes" Sep 16 13:36:37.753502 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:37.753464 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:39.076645 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.076613 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:39.469959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.469929 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" podUID="90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" containerName="main" containerID="cri-o://fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75" gracePeriod=45 Sep 16 13:36:39.723867 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.723792 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:36:39.805409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.805379 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tls-certs\") pod \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " Sep 16 13:36:39.805612 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.805446 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-home\") pod \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " Sep 16 13:36:39.805612 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.805472 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2mk9\" (UniqueName: \"kubernetes.io/projected/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-kube-api-access-z2mk9\") pod \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " Sep 16 13:36:39.805612 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.805490 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-dshm\") pod \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " Sep 16 13:36:39.805806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.805712 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-home" (OuterVolumeSpecName: "home") pod "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" (UID: "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:39.807718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.807692 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" (UID: "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:36:39.808007 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.807988 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-dshm" (OuterVolumeSpecName: "dshm") pod "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" (UID: "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:39.808094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.808005 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-kube-api-access-z2mk9" (OuterVolumeSpecName: "kube-api-access-z2mk9") pod "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" (UID: "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f"). InnerVolumeSpecName "kube-api-access-z2mk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:36:39.906708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906677 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-model-cache\") pod \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " Sep 16 13:36:39.906708 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906710 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tmp-dir\") pod \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\" (UID: \"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f\") " Sep 16 13:36:39.906905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906872 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tls-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:39.906905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906882 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-home\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:39.906905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906891 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2mk9\" (UniqueName: \"kubernetes.io/projected/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-kube-api-access-z2mk9\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:39.906905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906898 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-dshm\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:39.907037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.906925 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" (UID: "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:39.907070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:39.907028 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" (UID: "90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:40.007760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.007675 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-model-cache\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:40.007760 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.007698 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f-tmp-dir\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:36:40.008944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.008918 2581 generic.go:356] "Generic (PLEG): container finished" podID="90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" containerID="fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75" exitCode=0 Sep 16 13:36:40.009054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.008956 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" Sep 16 13:36:40.009054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.008953 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" event={"ID":"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f","Type":"ContainerDied","Data":"fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75"} Sep 16 13:36:40.009138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.009082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv" event={"ID":"90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f","Type":"ContainerDied","Data":"ee0ddc5e43e3a965a2a5fe109783863f706c17b148cfae697f85c748dd142fc4"} Sep 16 13:36:40.009138 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.009106 2581 scope.go:117] "RemoveContainer" containerID="fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75" Sep 16 13:36:40.018991 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.018970 2581 scope.go:117] "RemoveContainer" containerID="fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75" Sep 16 13:36:40.019242 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:36:40.019224 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75\": container with ID starting with fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75 not found: ID does not exist" containerID="fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75" Sep 16 13:36:40.019345 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.019253 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75"} err="failed to get container status \"fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75\": rpc error: code = NotFound desc = could not find container \"fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75\": container with ID starting with fdbc2117ee59dac861cb2e21f1ae08d3037bdf739c65e912da8032b476d58b75 not found: ID does not exist" Sep 16 13:36:40.044240 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.044215 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv"] Sep 16 13:36:40.049068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.049043 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f27ea8/distinct-2-w4-f27ea8-kserve-6f44967bf-tfmfv"] Sep 16 13:36:40.475390 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:40.475347 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:41.388265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:41.388225 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" path="/var/lib/kubelet/pods/90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f/volumes" Sep 16 13:36:41.428759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:41.428713 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:42.179536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:42.179501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:51.401951 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:51.401915 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:51.597761 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:51.597729 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:51.913501 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:51.913463 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:52.369444 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:52.369415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:52.560419 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:52.560385 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:55.550818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:55.550786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:55.700422 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:55.700387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:56.763554 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:56.763518 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:58.241166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:58.241075 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:59.177635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:59.177577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:59.296173 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:59.296146 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:59.443718 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:59.443620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:36:59.902937 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:36:59.902899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:00.008339 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:00.008305 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:00.441146 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:00.441107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:01.356148 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:01.356116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:03.373738 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:03.373703 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:03.502841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:03.502806 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:05.339465 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:05.339432 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:12.446395 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:12.446357 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:18.497502 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:18.497462 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:18.747814 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:18.747730 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:18.847420 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:18.847388 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:20.063196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:20.063152 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:21.732567 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:21.732531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:22.792606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:22.792567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:23.751315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:23.751272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:24.034599 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:24.034511 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx"] Sep 16 13:37:24.040913 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:24.040888 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c71ad7-q5lhx"] Sep 16 13:37:24.630366 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:24.630322 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:25.388871 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:25.388841 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fea35b3-1421-4dd8-b67b-407f4aaa315e" path="/var/lib/kubelet/pods/3fea35b3-1421-4dd8-b67b-407f4aaa315e/volumes" Sep 16 13:37:25.542206 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:25.542170 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:26.564620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:26.564572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:26.647819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:26.647785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:26.719311 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:26.719266 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm"] Sep 16 13:37:26.719519 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:26.719494 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" containerID="cri-o://8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0" gracePeriod=30 Sep 16 13:37:26.743915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:26.743884 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:27.001133 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:27.001101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:28.928754 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:28.928722 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:29.666299 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:29.666258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:29.804892 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:29.804845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:30.026862 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:30.026832 2581 patch_prober.go:28] interesting pod/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:37:30.027252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:30.026894 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:37:30.319915 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:30.319831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:31.350152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:31.350116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:31.972399 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:31.972368 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:37:32.098132 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098045 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-token\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098132 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098089 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-socket\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098132 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098121 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mcfx\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-kube-api-access-8mcfx\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098166 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efc3a8d-72a1-4177-991f-e24f53025a69-istiod-ca-cert\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098214 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-envoy\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098230 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-credential-socket\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098261 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-data\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098452 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:32.098716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098525 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:32.098716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098548 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-data" (OuterVolumeSpecName: "istio-data") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:32.098716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098581 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-podinfo\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098638 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-certs\") pod \"9efc3a8d-72a1-4177-991f-e24f53025a69\" (UID: \"9efc3a8d-72a1-4177-991f-e24f53025a69\") " Sep 16 13:37:32.098716 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098667 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efc3a8d-72a1-4177-991f-e24f53025a69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:37:32.099044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098842 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:32.099044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098981 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.099044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.098995 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.099044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.099009 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.099044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.099019 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efc3a8d-72a1-4177-991f-e24f53025a69-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.099044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.099029 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.100657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.100625 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-token" (OuterVolumeSpecName: "istio-token") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:37:32.100779 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.100671 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-kube-api-access-8mcfx" (OuterVolumeSpecName: "kube-api-access-8mcfx") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "kube-api-access-8mcfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:37:32.100841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.100821 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:37:32.100880 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.100821 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9efc3a8d-72a1-4177-991f-e24f53025a69" (UID: "9efc3a8d-72a1-4177-991f-e24f53025a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:32.163128 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.163093 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:32.200423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.200327 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.200423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.200362 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.200423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.200375 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.200423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.200397 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mcfx\" (UniqueName: \"kubernetes.io/projected/9efc3a8d-72a1-4177-991f-e24f53025a69-kube-api-access-8mcfx\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:37:32.336221 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.336191 2581 generic.go:356] "Generic (PLEG): container finished" podID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerID="8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0" exitCode=0 Sep 16 13:37:32.336424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.336240 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" Sep 16 13:37:32.336424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.336266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" event={"ID":"9efc3a8d-72a1-4177-991f-e24f53025a69","Type":"ContainerDied","Data":"8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0"} Sep 16 13:37:32.336424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.336334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm" event={"ID":"9efc3a8d-72a1-4177-991f-e24f53025a69","Type":"ContainerDied","Data":"da0857db8f14df0bfb1d5699eaa2152344161d9dbcf323aaaf8b13f4d108b0e8"} Sep 16 13:37:32.336424 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.336362 2581 scope.go:117] "RemoveContainer" containerID="8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0" Sep 16 13:37:32.346481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.346464 2581 scope.go:117] "RemoveContainer" containerID="7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da" Sep 16 13:37:32.355211 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.355192 2581 scope.go:117] "RemoveContainer" containerID="8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0" Sep 16 13:37:32.355486 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:37:32.355447 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0\": container with ID starting with 8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0 not found: ID does not exist" containerID="8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0" Sep 16 13:37:32.355486 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.355472 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0"} err="failed to get container status \"8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0\": rpc error: code = NotFound desc = could not find container \"8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0\": container with ID starting with 8cec5f15a5725c3b58f1edae439cd55ad0b1908b12478573607ea1844ec48ef0 not found: ID does not exist" Sep 16 13:37:32.355565 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.355495 2581 scope.go:117] "RemoveContainer" containerID="7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da" Sep 16 13:37:32.355721 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:37:32.355704 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da\": container with ID starting with 7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da not found: ID does not exist" containerID="7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da" Sep 16 13:37:32.355764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.355725 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da"} err="failed to get container status \"7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da\": rpc error: code = NotFound desc = could not find container \"7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da\": container with ID starting with 7305083e00ff1bde575ca6ab0dc58861efa5c0804bb80f3742ea35dc255cc3da not found: ID does not exist" Sep 16 13:37:32.361478 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.361454 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm"] Sep 16 13:37:32.364656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:32.364635 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a235ace-openshift-default-788d5dddd4-l2mfm"] Sep 16 13:37:33.387536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.387503 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" path="/var/lib/kubelet/pods/9efc3a8d-72a1-4177-991f-e24f53025a69/volumes" Sep 16 13:37:33.407984 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.407953 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:33.510490 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.510461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:33.622234 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.622207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:33.730165 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.730130 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:33.736390 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736363 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn"] Sep 16 13:37:33.736789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736778 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15883caa-b9ee-424b-a01d-a6d536e21db1" containerName="revoke-keys" Sep 16 13:37:33.736833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736791 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15883caa-b9ee-424b-a01d-a6d536e21db1" containerName="revoke-keys" Sep 16 13:37:33.736833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736803 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" Sep 16 13:37:33.736833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736808 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" Sep 16 13:37:33.736833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736815 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" Sep 16 13:37:33.736833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736820 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" Sep 16 13:37:33.736833 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736832 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" containerName="main" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736837 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" containerName="main" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736845 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60986dd7-b33f-4047-93f4-88f22f7db4f5" containerName="revoke-keys" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736850 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60986dd7-b33f-4047-93f4-88f22f7db4f5" containerName="revoke-keys" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736914 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="90dcbbbf-2b36-47a3-8f81-4b6eb4f3da0f" containerName="main" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736924 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60986dd7-b33f-4047-93f4-88f22f7db4f5" containerName="revoke-keys" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736931 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736939 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9efc3a8d-72a1-4177-991f-e24f53025a69" containerName="istio-proxy" Sep 16 13:37:33.737230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.736945 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15883caa-b9ee-424b-a01d-a6d536e21db1" containerName="revoke-keys" Sep 16 13:37:33.741686 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.741664 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:33.742264 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.742242 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj"] Sep 16 13:37:33.744028 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.744005 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8c3cdf-dockercfg-68vcm\"" Sep 16 13:37:33.744134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.744074 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8c3cdf\"" Sep 16 13:37:33.745635 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.745621 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.751647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.751555 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn"] Sep 16 13:37:33.757163 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.757144 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj"] Sep 16 13:37:33.817571 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.817536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb749133-4021-438e-b6f3-611bf776f67a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:33.817759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.817584 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g576s\" (UniqueName: \"kubernetes.io/projected/fb749133-4021-438e-b6f3-611bf776f67a-kube-api-access-g576s\") pod \"payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:33.817759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.817610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d893da2-e298-4a92-9f13-328202499b97-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.817759 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.817680 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbb8\" (UniqueName: \"kubernetes.io/projected/0d893da2-e298-4a92-9f13-328202499b97-kube-api-access-zkbb8\") pod \"payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.918812 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.918775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb749133-4021-438e-b6f3-611bf776f67a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:33.918996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.918827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g576s\" (UniqueName: \"kubernetes.io/projected/fb749133-4021-438e-b6f3-611bf776f67a-kube-api-access-g576s\") pod \"payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:33.918996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.918848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d893da2-e298-4a92-9f13-328202499b97-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.918996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.918872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbb8\" (UniqueName: \"kubernetes.io/projected/0d893da2-e298-4a92-9f13-328202499b97-kube-api-access-zkbb8\") pod \"payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.919785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.919761 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb749133-4021-438e-b6f3-611bf776f67a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:33.919838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.919761 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d893da2-e298-4a92-9f13-328202499b97-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.927489 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.927467 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbb8\" (UniqueName: \"kubernetes.io/projected/0d893da2-e298-4a92-9f13-328202499b97-kube-api-access-zkbb8\") pod \"payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:33.927607 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:33.927506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g576s\" (UniqueName: \"kubernetes.io/projected/fb749133-4021-438e-b6f3-611bf776f67a-kube-api-access-g576s\") pod \"payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:34.031433 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:34.031325 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:34.056068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:34.056046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:34.062986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:34.062969 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:34.348901 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:34.348869 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn"] Sep 16 13:37:34.349305 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:37:34.349266 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb749133_4021_438e_b6f3_611bf776f67a.slice/crio-9cfc9f3ee0203e5ddbc1946c266df74cedcf4379b87b38d16662fc6ed9a9db30 WatchSource:0}: Error finding container 9cfc9f3ee0203e5ddbc1946c266df74cedcf4379b87b38d16662fc6ed9a9db30: Status 404 returned error can't find the container with id 9cfc9f3ee0203e5ddbc1946c266df74cedcf4379b87b38d16662fc6ed9a9db30 Sep 16 13:37:34.356333 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:34.356310 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj"] Sep 16 13:37:34.357704 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:37:34.357682 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d893da2_e298_4a92_9f13_328202499b97.slice/crio-cc4adc2b46656a7b15c0487a72d30102881da8545be0ec58141a0f49ba6a5e50 WatchSource:0}: Error finding container cc4adc2b46656a7b15c0487a72d30102881da8545be0ec58141a0f49ba6a5e50: Status 404 returned error can't find the container with id cc4adc2b46656a7b15c0487a72d30102881da8545be0ec58141a0f49ba6a5e50 Sep 16 13:37:34.987365 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:34.987320 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:35.363343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.363247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" event={"ID":"0d893da2-e298-4a92-9f13-328202499b97","Type":"ContainerStarted","Data":"ba502b06d947fe238fbcb08e9670d0b9733483a18667269a57147d4352442ab8"} Sep 16 13:37:35.363343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.363336 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:35.363559 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.363367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" event={"ID":"0d893da2-e298-4a92-9f13-328202499b97","Type":"ContainerStarted","Data":"cc4adc2b46656a7b15c0487a72d30102881da8545be0ec58141a0f49ba6a5e50"} Sep 16 13:37:35.365038 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.365013 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" event={"ID":"fb749133-4021-438e-b6f3-611bf776f67a","Type":"ContainerStarted","Data":"3575c1f0f138abb1bb9812ba38a0bedf56f948d221de4b3a070810a34ae9d592"} Sep 16 13:37:35.365169 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.365050 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" event={"ID":"fb749133-4021-438e-b6f3-611bf776f67a","Type":"ContainerStarted","Data":"9cfc9f3ee0203e5ddbc1946c266df74cedcf4379b87b38d16662fc6ed9a9db30"} Sep 16 13:37:35.365169 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.365075 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:35.390144 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.390096 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" podStartSLOduration=2.39007915 podStartE2EDuration="2.39007915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:35.387996564 +0000 UTC m=+1740.507723130" watchObservedRunningTime="2026-09-16 13:37:35.39007915 +0000 UTC m=+1740.509805700" Sep 16 13:37:35.418853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.418807 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" podStartSLOduration=2.4187932979999998 podStartE2EDuration="2.418793298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:35.417050513 +0000 UTC m=+1740.536777083" watchObservedRunningTime="2026-09-16 13:37:35.418793298 +0000 UTC m=+1740.538519897" Sep 16 13:37:35.846017 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:35.845976 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:46.373468 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:46.373436 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:37:46.373861 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:46.373504 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:37:50.279574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:50.279534 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:50.700493 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:50.700458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:51.569707 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:51.569675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:52.839323 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:52.839261 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:53.850264 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:53.850229 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:54.805226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:54.805192 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:55.670735 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:55.670703 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:56.435611 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:56.435571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:57.444640 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:57.444604 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:58.259578 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:58.259544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:59.112118 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:59.112084 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:37:59.722097 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:37:59.722054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:03.926520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:03.926479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:11.047694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:11.047656 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:17.604168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:17.604136 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:31.197423 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:31.197381 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:31.515111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:31.515076 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:35.517912 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:35.517885 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:38:35.521037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:35.521018 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:38:35.727294 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:35.727260 2581 scope.go:117] "RemoveContainer" containerID="a77e1d4725ca18681f19f7775f0d6aec8dca1b3c2e881dc4ff2377a3e82d9570" Sep 16 13:38:53.218363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:53.218327 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:38:53.300425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:38:53.300387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:41.302513 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:41.302479 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5"] Sep 16 13:39:41.303072 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:41.302703 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" containerID="cri-o://7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2" gracePeriod=30 Sep 16 13:39:41.334375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:41.334346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:41.545374 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:41.545339 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:42.885472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:42.885438 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 13:39:42.885877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:42.885507 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 13:39:44.027130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:44.027093 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624"] Sep 16 13:39:44.030019 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:44.029990 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a21a40-pc624"] Sep 16 13:39:45.393702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:45.393661 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df13f358-1ef6-4b3d-8094-8686fb83e8b3" path="/var/lib/kubelet/pods/df13f358-1ef6-4b3d-8094-8686fb83e8b3/volumes" Sep 16 13:39:46.563594 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.563570 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:39:46.631083 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631012 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-socket\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631083 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631071 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98218b1f-f697-4fbb-960d-bb45b6f38558-istiod-ca-cert\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631097 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g22s6\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-kube-api-access-g22s6\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631121 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-token\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631146 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-credential-socket\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631209 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-certs\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631244 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-data\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631296 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-podinfo\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631330 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:39:46.631628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631337 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-envoy\") pod \"98218b1f-f697-4fbb-960d-bb45b6f38558\" (UID: \"98218b1f-f697-4fbb-960d-bb45b6f38558\") " Sep 16 13:39:46.631628 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631579 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98218b1f-f697-4fbb-960d-bb45b6f38558-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:39:46.631793 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631757 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.631793 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631777 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98218b1f-f697-4fbb-960d-bb45b6f38558-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.631903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631835 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:39:46.631903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631846 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-data" (OuterVolumeSpecName: "istio-data") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:39:46.632021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.631950 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:39:46.633522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.633493 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:39:46.633627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.633601 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-kube-api-access-g22s6" (OuterVolumeSpecName: "kube-api-access-g22s6") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "kube-api-access-g22s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:39:46.633627 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.633613 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-token" (OuterVolumeSpecName: "istio-token") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:39:46.633818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.633800 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98218b1f-f697-4fbb-960d-bb45b6f38558" (UID: "98218b1f-f697-4fbb-960d-bb45b6f38558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:39:46.732991 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.732963 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22s6\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-kube-api-access-g22s6\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.732991 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.732987 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.732991 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.732997 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.733182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.733005 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.733182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.733014 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.733182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.733021 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:46.733182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:46.733028 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98218b1f-f697-4fbb-960d-bb45b6f38558-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:39:47.182092 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.182051 2581 generic.go:356] "Generic (PLEG): container finished" podID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerID="7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2" exitCode=0 Sep 16 13:39:47.182255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.182102 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" Sep 16 13:39:47.182255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.182132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" event={"ID":"98218b1f-f697-4fbb-960d-bb45b6f38558","Type":"ContainerDied","Data":"7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2"} Sep 16 13:39:47.182255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.182190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5" event={"ID":"98218b1f-f697-4fbb-960d-bb45b6f38558","Type":"ContainerDied","Data":"628be1255928000d612caee390be9dedce24f5a9ce4c3dbf0d47e6511f5f2c37"} Sep 16 13:39:47.182255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.182225 2581 scope.go:117] "RemoveContainer" containerID="7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2" Sep 16 13:39:47.193350 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.193322 2581 scope.go:117] "RemoveContainer" containerID="bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158" Sep 16 13:39:47.206347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.206327 2581 scope.go:117] "RemoveContainer" containerID="7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2" Sep 16 13:39:47.206621 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:39:47.206603 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2\": container with ID starting with 7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2 not found: ID does not exist" containerID="7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2" Sep 16 13:39:47.206690 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.206628 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2"} err="failed to get container status \"7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2\": rpc error: code = NotFound desc = could not find container \"7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2\": container with ID starting with 7361bec5e3c2fb597de9c9ceb879ff220c9710d74455a28aaafcbfbbeae68cb2 not found: ID does not exist" Sep 16 13:39:47.206690 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.206648 2581 scope.go:117] "RemoveContainer" containerID="bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158" Sep 16 13:39:47.206912 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:39:47.206892 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158\": container with ID starting with bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158 not found: ID does not exist" containerID="bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158" Sep 16 13:39:47.206971 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.206923 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158"} err="failed to get container status \"bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158\": rpc error: code = NotFound desc = could not find container \"bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158\": container with ID starting with bf2a96621d88bfbff4c77487e8a508e372e70a6278ec3b96b213fe5ddc519158 not found: ID does not exist" Sep 16 13:39:47.208021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.208000 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5"] Sep 16 13:39:47.212422 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.212397 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a21a40-openshift-default-59bc8b5db6-hmjv5"] Sep 16 13:39:47.388148 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:47.388116 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" path="/var/lib/kubelet/pods/98218b1f-f697-4fbb-960d-bb45b6f38558/volumes" Sep 16 13:39:49.901678 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:49.901627 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:39:56.050875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:39:56.050836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:28.526545 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:40:28.526505 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:32.302268 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:40:32.302216 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:32.410222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:40:32.410181 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:40:32.528325 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:40:32.528274 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:06.139772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:06.139734 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:06.248777 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:06.248743 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:07.836242 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:07.836199 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:20.174472 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:20.174437 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:20.375875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:20.375842 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:41:24.029453 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:24.029412 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z"] Sep 16 13:41:24.034368 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:24.034341 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f27ea8-lg89z"] Sep 16 13:41:25.388806 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:25.388764 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60986dd7-b33f-4047-93f4-88f22f7db4f5" path="/var/lib/kubelet/pods/60986dd7-b33f-4047-93f4-88f22f7db4f5/volumes" Sep 16 13:41:35.774848 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:35.774815 2581 scope.go:117] "RemoveContainer" containerID="cf562767cc8abfeb07952e3df8b4b63688ae6c62e4d0724284a6374cf53f80f4" Sep 16 13:41:38.936252 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:41:38.936219 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:01.404267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:01.404227 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:21.002409 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:21.002372 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:21.281695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:21.281612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:22.039679 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:22.039643 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:23.028667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:23.028634 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:23.733073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:23.733028 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:24.639615 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:24.639576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:25.474396 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:25.474359 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:26.322139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:26.322104 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:35.797565 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:35.797537 2581 scope.go:117] "RemoveContainer" containerID="6f18f3e2f699483bee0221518ef1963148114764f8744c86f6c833482d502ff2" Sep 16 13:42:35.806679 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:35.806658 2581 scope.go:117] "RemoveContainer" containerID="200c904f7e191f46435859e5c7dffb6b7ddd35e36f894899520cff0c2d5a1b6b" Sep 16 13:42:36.515366 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:36.515316 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:42.683583 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:42.683546 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:42.860967 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:42.860937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:43.319154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:43.319111 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:43.411069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:43.411031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:48.764662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:48.764624 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:53.941021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:53.940981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:54.109548 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:54.109509 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:42:54.667220 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:42:54.667180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:07.583369 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:07.583270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:15.424488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:15.424450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:21.726959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:21.726921 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:21.825076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:21.825036 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:22.179794 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:22.179716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:22.635410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:22.635359 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:23.090769 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:23.090732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:25.301247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:25.301205 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:25.996017 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:25.995977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:26.163301 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:26.163246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:26.330405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:26.330303 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:26.860807 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:26.860774 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:28.398209 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:28.398168 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:30.061210 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:30.061175 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:31.304838 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:31.304806 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.764328 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:32.764229 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:32.896989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:32.896947 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:33.083875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.083782 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:33.444606 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.444525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:33.602484 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.602452 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd"] Sep 16 13:43:33.602867 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.602855 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" Sep 16 13:43:33.602913 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.602869 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" Sep 16 13:43:33.602952 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.602946 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" Sep 16 13:43:33.602987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.602956 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" Sep 16 13:43:33.606053 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.606036 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.609068 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.609050 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-744b19\"" Sep 16 13:43:33.645919 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.645889 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd"] Sep 16 13:43:33.687351 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.687318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:33.750371 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.750341 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl"] Sep 16 13:43:33.750818 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.750804 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" Sep 16 13:43:33.750868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.750821 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98218b1f-f697-4fbb-960d-bb45b6f38558" containerName="istio-proxy" Sep 16 13:43:33.753988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.753973 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:33.759178 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.759160 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-744b19-dockercfg-4rrlc\"" Sep 16 13:43:33.770082 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.770056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq4h\" (UniqueName: \"kubernetes.io/projected/0aaa380f-14b9-4af7-aba7-696a887d833c-kube-api-access-swq4h\") pod \"payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.770536 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.770511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaa380f-14b9-4af7-aba7-696a887d833c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.775267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.775243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl"] Sep 16 13:43:33.871263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.871226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaa380f-14b9-4af7-aba7-696a887d833c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.871479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.871303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673a0ee8-2662-41ab-8141-c00e7cf0e52d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:33.871479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.871337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244kk\" (UniqueName: \"kubernetes.io/projected/673a0ee8-2662-41ab-8141-c00e7cf0e52d-kube-api-access-244kk\") pod \"payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:33.871479 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.871365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq4h\" (UniqueName: \"kubernetes.io/projected/0aaa380f-14b9-4af7-aba7-696a887d833c-kube-api-access-swq4h\") pod \"payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.871987 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.871969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaa380f-14b9-4af7-aba7-696a887d833c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.891706 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.891674 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq4h\" (UniqueName: \"kubernetes.io/projected/0aaa380f-14b9-4af7-aba7-696a887d833c-kube-api-access-swq4h\") pod \"payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.916049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.916024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:33.973687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.973032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673a0ee8-2662-41ab-8141-c00e7cf0e52d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:33.973687 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.973085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244kk\" (UniqueName: \"kubernetes.io/projected/673a0ee8-2662-41ab-8141-c00e7cf0e52d-kube-api-access-244kk\") pod \"payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:33.974458 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.974405 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673a0ee8-2662-41ab-8141-c00e7cf0e52d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:33.987414 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:33.987384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-244kk\" (UniqueName: \"kubernetes.io/projected/673a0ee8-2662-41ab-8141-c00e7cf0e52d-kube-api-access-244kk\") pod \"payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:34.064977 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.064895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:34.220090 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:43:34.220060 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aaa380f_14b9_4af7_aba7_696a887d833c.slice/crio-53e482a1c351350ef414c02e37673eab41751a57b4f9875e513ea1917e466a82 WatchSource:0}: Error finding container 53e482a1c351350ef414c02e37673eab41751a57b4f9875e513ea1917e466a82: Status 404 returned error can't find the container with id 53e482a1c351350ef414c02e37673eab41751a57b4f9875e513ea1917e466a82 Sep 16 13:43:34.220894 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.220872 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd"] Sep 16 13:43:34.276339 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.276313 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:34.372483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.372449 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl"] Sep 16 13:43:34.374944 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:43:34.374913 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673a0ee8_2662_41ab_8141_c00e7cf0e52d.slice/crio-90ee514b8db5202b17b3e4e3af47e039d3c9349db1837492aa5a9d93fe4b9adb WatchSource:0}: Error finding container 90ee514b8db5202b17b3e4e3af47e039d3c9349db1837492aa5a9d93fe4b9adb: Status 404 returned error can't find the container with id 90ee514b8db5202b17b3e4e3af47e039d3c9349db1837492aa5a9d93fe4b9adb Sep 16 13:43:34.542070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.542026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" event={"ID":"0aaa380f-14b9-4af7-aba7-696a887d833c","Type":"ContainerStarted","Data":"cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3"} Sep 16 13:43:34.542247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.542094 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" event={"ID":"0aaa380f-14b9-4af7-aba7-696a887d833c","Type":"ContainerStarted","Data":"53e482a1c351350ef414c02e37673eab41751a57b4f9875e513ea1917e466a82"} Sep 16 13:43:34.542247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.542124 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:34.543772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.543745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" event={"ID":"673a0ee8-2662-41ab-8141-c00e7cf0e52d","Type":"ContainerStarted","Data":"83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f"} Sep 16 13:43:34.543899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.543791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" event={"ID":"673a0ee8-2662-41ab-8141-c00e7cf0e52d","Type":"ContainerStarted","Data":"90ee514b8db5202b17b3e4e3af47e039d3c9349db1837492aa5a9d93fe4b9adb"} Sep 16 13:43:34.543899 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.543812 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:34.580681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.580630 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" podStartSLOduration=1.580615833 podStartE2EDuration="1.580615833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:34.578422722 +0000 UTC m=+2099.698149270" watchObservedRunningTime="2026-09-16 13:43:34.580615833 +0000 UTC m=+2099.700342412" Sep 16 13:43:34.622523 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:34.622426 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" podStartSLOduration=1.6224064870000001 podStartE2EDuration="1.622406487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:34.618652071 +0000 UTC m=+2099.738378620" watchObservedRunningTime="2026-09-16 13:43:34.622406487 +0000 UTC m=+2099.742133034" Sep 16 13:43:35.556689 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:35.556660 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:43:35.560938 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:35.560919 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:43:35.933150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:35.933071 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:37.714087 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:37.714052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:45.552289 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:45.552259 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:43:45.552662 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:45.552340 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:43:56.365819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:56.365783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:56.864568 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:56.864532 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.037578 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:57.037544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.399789 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:57.399746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:57.581466 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:57.581422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:43:58.845694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:43:58.845660 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:00.599254 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:00.599221 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.640523 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:01.640481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:01.902855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:01.902755 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:03.087908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:03.087864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:04.746488 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:04.746454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:06.222858 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:06.222822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:07.363066 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:07.363021 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:07.475721 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:07.475688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:07.619896 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:07.619803 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.238493 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.238454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.450416 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.450373 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:08.517196 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.517126 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8"] Sep 16 13:44:08.521037 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.521020 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.526748 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.526725 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-804072\"" Sep 16 13:44:08.528869 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.528852 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-804072-dockercfg-pbwmt\"" Sep 16 13:44:08.542326 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.542302 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8"] Sep 16 13:44:08.583758 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.583718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0462f3b3-2002-4991-967d-6028145dfdb6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-804072-5b6788555f-hlvd8\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.583946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.583900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64rc\" (UniqueName: \"kubernetes.io/projected/0462f3b3-2002-4991-967d-6028145dfdb6-kube-api-access-x64rc\") pod \"payload-processing-e2e-api-b-804072-5b6788555f-hlvd8\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.658967 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.658926 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx"] Sep 16 13:44:08.662622 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.662601 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.685389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.685365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0462f3b3-2002-4991-967d-6028145dfdb6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-804072-5b6788555f-hlvd8\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.685499 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.685449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64rc\" (UniqueName: \"kubernetes.io/projected/0462f3b3-2002-4991-967d-6028145dfdb6-kube-api-access-x64rc\") pod \"payload-processing-e2e-api-b-804072-5b6788555f-hlvd8\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.686024 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.686000 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0462f3b3-2002-4991-967d-6028145dfdb6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-804072-5b6788555f-hlvd8\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.722786 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.722751 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64rc\" (UniqueName: \"kubernetes.io/projected/0462f3b3-2002-4991-967d-6028145dfdb6-kube-api-access-x64rc\") pod \"payload-processing-e2e-api-b-804072-5b6788555f-hlvd8\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.724533 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.724512 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx"] Sep 16 13:44:08.786699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.786626 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrtm\" (UniqueName: \"kubernetes.io/projected/6bef7c7d-033d-4383-8685-aa5c9ceb979d-kube-api-access-dbrtm\") pod \"payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.786699 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.786661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bef7c7d-033d-4383-8685-aa5c9ceb979d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.831675 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.831647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:08.888347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.888244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrtm\" (UniqueName: \"kubernetes.io/projected/6bef7c7d-033d-4383-8685-aa5c9ceb979d-kube-api-access-dbrtm\") pod \"payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.888347 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.888340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bef7c7d-033d-4383-8685-aa5c9ceb979d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.889520 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.889293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bef7c7d-033d-4383-8685-aa5c9ceb979d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.914198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.914152 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrtm\" (UniqueName: \"kubernetes.io/projected/6bef7c7d-033d-4383-8685-aa5c9ceb979d-kube-api-access-dbrtm\") pod \"payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:08.972549 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:08.972517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:09.136834 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.136798 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8"] Sep 16 13:44:09.138557 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:44:09.138526 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0462f3b3_2002_4991_967d_6028145dfdb6.slice/crio-8025e6d8b5944a4d225b1405d094695c3666e173566136e7640f2fc1c38ed71a WatchSource:0}: Error finding container 8025e6d8b5944a4d225b1405d094695c3666e173566136e7640f2fc1c38ed71a: Status 404 returned error can't find the container with id 8025e6d8b5944a4d225b1405d094695c3666e173566136e7640f2fc1c38ed71a Sep 16 13:44:09.285600 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:44:09.285566 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bef7c7d_033d_4383_8685_aa5c9ceb979d.slice/crio-edc50b02b22476ffb9941f69bea14f28710bb186e02235fbc89488a331b39bd9 WatchSource:0}: Error finding container edc50b02b22476ffb9941f69bea14f28710bb186e02235fbc89488a331b39bd9: Status 404 returned error can't find the container with id edc50b02b22476ffb9941f69bea14f28710bb186e02235fbc89488a331b39bd9 Sep 16 13:44:09.289421 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.289392 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx"] Sep 16 13:44:09.364877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.364839 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:09.764343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.764309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" event={"ID":"6bef7c7d-033d-4383-8685-aa5c9ceb979d","Type":"ContainerStarted","Data":"09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0"} Sep 16 13:44:09.764785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.764373 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" event={"ID":"6bef7c7d-033d-4383-8685-aa5c9ceb979d","Type":"ContainerStarted","Data":"edc50b02b22476ffb9941f69bea14f28710bb186e02235fbc89488a331b39bd9"} Sep 16 13:44:09.764785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.764445 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:09.766198 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.766172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" event={"ID":"0462f3b3-2002-4991-967d-6028145dfdb6","Type":"ContainerStarted","Data":"45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d"} Sep 16 13:44:09.766342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.766214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" event={"ID":"0462f3b3-2002-4991-967d-6028145dfdb6","Type":"ContainerStarted","Data":"8025e6d8b5944a4d225b1405d094695c3666e173566136e7640f2fc1c38ed71a"} Sep 16 13:44:09.766342 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.766317 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:09.805990 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.805932 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" podStartSLOduration=1.8059121500000002 podStartE2EDuration="1.80591215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:09.797892706 +0000 UTC m=+2134.917619257" watchObservedRunningTime="2026-09-16 13:44:09.80591215 +0000 UTC m=+2134.925638709" Sep 16 13:44:09.826860 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:09.826817 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" podStartSLOduration=1.8268025319999999 podStartE2EDuration="1.826802532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:09.82592759 +0000 UTC m=+2134.945654141" watchObservedRunningTime="2026-09-16 13:44:09.826802532 +0000 UTC m=+2134.946529143" Sep 16 13:44:10.437255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:10.437224 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:11.946722 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:11.946684 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:13.733646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:13.733612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:15.147620 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:15.147584 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:18.512325 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:18.512265 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:19.994618 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:19.994581 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:20.775358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:20.775325 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:44:20.775543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:20.775403 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:44:20.998867 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:20.998828 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:21.276217 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:21.276177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:22.122936 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:22.122904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:23.179369 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:23.179327 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.492530 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.492496 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d"] Sep 16 13:44:24.496534 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.496508 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.501929 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.501906 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:44:24.502486 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.502467 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:44:24.503081 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.503061 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:44:24.585215 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.585184 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d"] Sep 16 13:44:24.601879 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.601834 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:24.644591 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.644559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8p6k\" (UniqueName: \"kubernetes.io/projected/752e346c-8660-42a0-b606-5b71a947ab70-kube-api-access-z8p6k\") pod \"maas-api-revoke-keys-e2e-api-a-744b19-xpx5d\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.644726 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.644618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752e346c-8660-42a0-b606-5b71a947ab70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-744b19-xpx5d\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.745402 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.745322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8p6k\" (UniqueName: \"kubernetes.io/projected/752e346c-8660-42a0-b606-5b71a947ab70-kube-api-access-z8p6k\") pod \"maas-api-revoke-keys-e2e-api-a-744b19-xpx5d\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.745402 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.745386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752e346c-8660-42a0-b606-5b71a947ab70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-744b19-xpx5d\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.745974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.745954 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752e346c-8660-42a0-b606-5b71a947ab70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-744b19-xpx5d\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.759724 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.759695 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8p6k\" (UniqueName: \"kubernetes.io/projected/752e346c-8660-42a0-b606-5b71a947ab70-kube-api-access-z8p6k\") pod \"maas-api-revoke-keys-e2e-api-a-744b19-xpx5d\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:24.807317 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:24.807269 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:25.085329 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:44:25.085296 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752e346c_8660_42a0_b606_5b71a947ab70.slice/crio-8b3e99cf632846096137e2c87b9fc37b7e3d718410a155300dff283cb81b4019 WatchSource:0}: Error finding container 8b3e99cf632846096137e2c87b9fc37b7e3d718410a155300dff283cb81b4019: Status 404 returned error can't find the container with id 8b3e99cf632846096137e2c87b9fc37b7e3d718410a155300dff283cb81b4019 Sep 16 13:44:25.085512 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:25.085368 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d"] Sep 16 13:44:25.784183 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:25.784143 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:25.878377 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:25.878334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" event={"ID":"752e346c-8660-42a0-b606-5b71a947ab70","Type":"ContainerStarted","Data":"72ffdbf717803236a44a2b25d1c99a544daaa664db6a8554377b4a1d77a1dba9"} Sep 16 13:44:25.878558 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:25.878434 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" event={"ID":"752e346c-8660-42a0-b606-5b71a947ab70","Type":"ContainerStarted","Data":"8b3e99cf632846096137e2c87b9fc37b7e3d718410a155300dff283cb81b4019"} Sep 16 13:44:25.901914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:25.901867 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" podStartSLOduration=1.901851237 podStartE2EDuration="1.901851237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:25.90084544 +0000 UTC m=+2151.020572021" watchObservedRunningTime="2026-09-16 13:44:25.901851237 +0000 UTC m=+2151.021577785" Sep 16 13:44:26.886505 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:26.886471 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:26.887080 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:26.886727 2581 generic.go:356] "Generic (PLEG): container finished" podID="752e346c-8660-42a0-b606-5b71a947ab70" containerID="72ffdbf717803236a44a2b25d1c99a544daaa664db6a8554377b4a1d77a1dba9" exitCode=0 Sep 16 13:44:26.887080 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:26.886798 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" event={"ID":"752e346c-8660-42a0-b606-5b71a947ab70","Type":"ContainerDied","Data":"72ffdbf717803236a44a2b25d1c99a544daaa664db6a8554377b4a1d77a1dba9"} Sep 16 13:44:28.035630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.035604 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:28.176933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.176853 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8p6k\" (UniqueName: \"kubernetes.io/projected/752e346c-8660-42a0-b606-5b71a947ab70-kube-api-access-z8p6k\") pod \"752e346c-8660-42a0-b606-5b71a947ab70\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " Sep 16 13:44:28.176933 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.176934 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752e346c-8660-42a0-b606-5b71a947ab70-maas-api-service-ca\") pod \"752e346c-8660-42a0-b606-5b71a947ab70\" (UID: \"752e346c-8660-42a0-b606-5b71a947ab70\") " Sep 16 13:44:28.177351 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.177306 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752e346c-8660-42a0-b606-5b71a947ab70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "752e346c-8660-42a0-b606-5b71a947ab70" (UID: "752e346c-8660-42a0-b606-5b71a947ab70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:28.179128 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.179101 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752e346c-8660-42a0-b606-5b71a947ab70-kube-api-access-z8p6k" (OuterVolumeSpecName: "kube-api-access-z8p6k") pod "752e346c-8660-42a0-b606-5b71a947ab70" (UID: "752e346c-8660-42a0-b606-5b71a947ab70"). InnerVolumeSpecName "kube-api-access-z8p6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:28.277729 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.277701 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8p6k\" (UniqueName: \"kubernetes.io/projected/752e346c-8660-42a0-b606-5b71a947ab70-kube-api-access-z8p6k\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:28.277729 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.277726 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/752e346c-8660-42a0-b606-5b71a947ab70-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:28.298176 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.298143 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:28.900121 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.900093 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" Sep 16 13:44:28.900318 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.900090 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d" event={"ID":"752e346c-8660-42a0-b606-5b71a947ab70","Type":"ContainerDied","Data":"8b3e99cf632846096137e2c87b9fc37b7e3d718410a155300dff283cb81b4019"} Sep 16 13:44:28.900318 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:28.900225 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3e99cf632846096137e2c87b9fc37b7e3d718410a155300dff283cb81b4019" Sep 16 13:44:29.476145 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:29.476111 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:30.715126 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:30.715082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:31.637810 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:31.637770 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:32.595318 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:32.595266 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:33.397985 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:33.397945 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:34.500060 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:34.500027 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:34.631401 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:34.631369 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:39.913364 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:39.913311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:40.410273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.410238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:40.439443 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.439405 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl"] Sep 16 13:44:40.439677 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.439649 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" podUID="673a0ee8-2662-41ab-8141-c00e7cf0e52d" containerName="payload-processing" containerID="cri-o://83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f" gracePeriod=30 Sep 16 13:44:40.453242 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.453201 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd"] Sep 16 13:44:40.453524 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.453499 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" podUID="0aaa380f-14b9-4af7-aba7-696a887d833c" containerName="payload-pre-processing" containerID="cri-o://cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3" gracePeriod=30 Sep 16 13:44:40.506904 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.506870 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:40.612878 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.612842 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:40.652299 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.652203 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d"] Sep 16 13:44:40.657095 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.657063 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-744b19-xpx5d"] Sep 16 13:44:40.707617 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.707591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:40.734159 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.734140 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:44:40.736969 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.736950 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:44:40.791189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.791157 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaa380f-14b9-4af7-aba7-696a887d833c-plugins-config-volume\") pod \"0aaa380f-14b9-4af7-aba7-696a887d833c\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " Sep 16 13:44:40.791405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.791206 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673a0ee8-2662-41ab-8141-c00e7cf0e52d-plugins-config-volume\") pod \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " Sep 16 13:44:40.791405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.791354 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-244kk\" (UniqueName: \"kubernetes.io/projected/673a0ee8-2662-41ab-8141-c00e7cf0e52d-kube-api-access-244kk\") pod \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\" (UID: \"673a0ee8-2662-41ab-8141-c00e7cf0e52d\") " Sep 16 13:44:40.791405 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.791392 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swq4h\" (UniqueName: \"kubernetes.io/projected/0aaa380f-14b9-4af7-aba7-696a887d833c-kube-api-access-swq4h\") pod \"0aaa380f-14b9-4af7-aba7-696a887d833c\" (UID: \"0aaa380f-14b9-4af7-aba7-696a887d833c\") " Sep 16 13:44:40.792030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.791600 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aaa380f-14b9-4af7-aba7-696a887d833c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0aaa380f-14b9-4af7-aba7-696a887d833c" (UID: "0aaa380f-14b9-4af7-aba7-696a887d833c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:40.792030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.791732 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aaa380f-14b9-4af7-aba7-696a887d833c-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:40.792460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.792425 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673a0ee8-2662-41ab-8141-c00e7cf0e52d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "673a0ee8-2662-41ab-8141-c00e7cf0e52d" (UID: "673a0ee8-2662-41ab-8141-c00e7cf0e52d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:40.794649 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.794622 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaa380f-14b9-4af7-aba7-696a887d833c-kube-api-access-swq4h" (OuterVolumeSpecName: "kube-api-access-swq4h") pod "0aaa380f-14b9-4af7-aba7-696a887d833c" (UID: "0aaa380f-14b9-4af7-aba7-696a887d833c"). InnerVolumeSpecName "kube-api-access-swq4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:40.795168 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.795142 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673a0ee8-2662-41ab-8141-c00e7cf0e52d-kube-api-access-244kk" (OuterVolumeSpecName: "kube-api-access-244kk") pod "673a0ee8-2662-41ab-8141-c00e7cf0e52d" (UID: "673a0ee8-2662-41ab-8141-c00e7cf0e52d"). InnerVolumeSpecName "kube-api-access-244kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:40.892456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.892423 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-244kk\" (UniqueName: \"kubernetes.io/projected/673a0ee8-2662-41ab-8141-c00e7cf0e52d-kube-api-access-244kk\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:40.892456 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.892454 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swq4h\" (UniqueName: \"kubernetes.io/projected/0aaa380f-14b9-4af7-aba7-696a887d833c-kube-api-access-swq4h\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:40.892656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.892466 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673a0ee8-2662-41ab-8141-c00e7cf0e52d-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:40.973773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.973687 2581 generic.go:356] "Generic (PLEG): container finished" podID="0aaa380f-14b9-4af7-aba7-696a887d833c" containerID="cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3" exitCode=0 Sep 16 13:44:40.973773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.973734 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" Sep 16 13:44:40.974272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.973778 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" event={"ID":"0aaa380f-14b9-4af7-aba7-696a887d833c","Type":"ContainerDied","Data":"cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3"} Sep 16 13:44:40.974272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.973834 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd" event={"ID":"0aaa380f-14b9-4af7-aba7-696a887d833c","Type":"ContainerDied","Data":"53e482a1c351350ef414c02e37673eab41751a57b4f9875e513ea1917e466a82"} Sep 16 13:44:40.974272 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.973863 2581 scope.go:117] "RemoveContainer" containerID="cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3" Sep 16 13:44:40.975141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.975119 2581 generic.go:356] "Generic (PLEG): container finished" podID="673a0ee8-2662-41ab-8141-c00e7cf0e52d" containerID="83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f" exitCode=0 Sep 16 13:44:40.975241 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.975151 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" event={"ID":"673a0ee8-2662-41ab-8141-c00e7cf0e52d","Type":"ContainerDied","Data":"83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f"} Sep 16 13:44:40.975241 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.975170 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" Sep 16 13:44:40.975241 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.975191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl" event={"ID":"673a0ee8-2662-41ab-8141-c00e7cf0e52d","Type":"ContainerDied","Data":"90ee514b8db5202b17b3e4e3af47e039d3c9349db1837492aa5a9d93fe4b9adb"} Sep 16 13:44:40.984005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.983987 2581 scope.go:117] "RemoveContainer" containerID="cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3" Sep 16 13:44:40.984369 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:44:40.984341 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3\": container with ID starting with cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3 not found: ID does not exist" containerID="cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3" Sep 16 13:44:40.984498 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.984377 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3"} err="failed to get container status \"cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3\": rpc error: code = NotFound desc = could not find container \"cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3\": container with ID starting with cefecac301aa25cee64e4d2561599c8edcc151bade5c5c6580572c4e88f02db3 not found: ID does not exist" Sep 16 13:44:40.984498 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.984400 2581 scope.go:117] "RemoveContainer" containerID="83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f" Sep 16 13:44:41.000919 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:40.993384 2581 scope.go:117] "RemoveContainer" containerID="83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f" Sep 16 13:44:41.001276 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:44:41.001253 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f\": container with ID starting with 83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f not found: ID does not exist" containerID="83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f" Sep 16 13:44:41.001399 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.001298 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f"} err="failed to get container status \"83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f\": rpc error: code = NotFound desc = could not find container \"83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f\": container with ID starting with 83f1475b80a9bcbea855c042f73328e8ee6729ec5b1ced9ce28569760510811f not found: ID does not exist" Sep 16 13:44:41.006655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.006632 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl"] Sep 16 13:44:41.011346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.011325 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-744b19-84c99747b9-p6vhl"] Sep 16 13:44:41.029379 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.029360 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd"] Sep 16 13:44:41.034346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.034327 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-744b19-7cd6c4dd78-7cjsd"] Sep 16 13:44:41.073961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.073936 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:41.168175 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.168141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:41.387860 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.387763 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aaa380f-14b9-4af7-aba7-696a887d833c" path="/var/lib/kubelet/pods/0aaa380f-14b9-4af7-aba7-696a887d833c/volumes" Sep 16 13:44:41.388245 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.388225 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673a0ee8-2662-41ab-8141-c00e7cf0e52d" path="/var/lib/kubelet/pods/673a0ee8-2662-41ab-8141-c00e7cf0e52d/volumes" Sep 16 13:44:41.388653 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.388633 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="752e346c-8660-42a0-b606-5b71a947ab70" path="/var/lib/kubelet/pods/752e346c-8660-42a0-b606-5b71a947ab70/volumes" Sep 16 13:44:41.679968 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:41.679882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:42.156955 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:42.156916 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:42.984996 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:42.984949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:43.839396 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:43.839361 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:44.836267 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:44.836232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:45.577743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:45.577704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:46.516039 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:46.516000 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:47.703182 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:47.703143 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:47.805766 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:47.805730 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:47.995214 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:47.995131 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:48.356753 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.356658 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c"] Sep 16 13:44:48.357195 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357176 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aaa380f-14b9-4af7-aba7-696a887d833c" containerName="payload-pre-processing" Sep 16 13:44:48.357329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357198 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aaa380f-14b9-4af7-aba7-696a887d833c" containerName="payload-pre-processing" Sep 16 13:44:48.357329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357231 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673a0ee8-2662-41ab-8141-c00e7cf0e52d" containerName="payload-processing" Sep 16 13:44:48.357329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357239 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="673a0ee8-2662-41ab-8141-c00e7cf0e52d" containerName="payload-processing" Sep 16 13:44:48.357329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357253 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752e346c-8660-42a0-b606-5b71a947ab70" containerName="revoke-keys" Sep 16 13:44:48.357329 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357261 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="752e346c-8660-42a0-b606-5b71a947ab70" containerName="revoke-keys" Sep 16 13:44:48.357593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357387 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="673a0ee8-2662-41ab-8141-c00e7cf0e52d" containerName="payload-processing" Sep 16 13:44:48.357593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357400 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aaa380f-14b9-4af7-aba7-696a887d833c" containerName="payload-pre-processing" Sep 16 13:44:48.357593 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.357408 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="752e346c-8660-42a0-b606-5b71a947ab70" containerName="revoke-keys" Sep 16 13:44:48.362181 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.362161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.364719 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.364696 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:44:48.365754 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.365722 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:44:48.365754 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.365740 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:44:48.369599 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.369572 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c"] Sep 16 13:44:48.464720 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.464685 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e258c7-59db-4802-a1cf-e301ece299d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-804072-plz9c\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.464891 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.464737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4cs\" (UniqueName: \"kubernetes.io/projected/d2e258c7-59db-4802-a1cf-e301ece299d9-kube-api-access-mz4cs\") pod \"maas-api-revoke-keys-e2e-api-b-804072-plz9c\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.550297 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.550241 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:48.565866 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.565835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e258c7-59db-4802-a1cf-e301ece299d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-804072-plz9c\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.566017 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.565897 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4cs\" (UniqueName: \"kubernetes.io/projected/d2e258c7-59db-4802-a1cf-e301ece299d9-kube-api-access-mz4cs\") pod \"maas-api-revoke-keys-e2e-api-b-804072-plz9c\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.566626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.566601 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e258c7-59db-4802-a1cf-e301ece299d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-804072-plz9c\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.574082 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.574061 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4cs\" (UniqueName: \"kubernetes.io/projected/d2e258c7-59db-4802-a1cf-e301ece299d9-kube-api-access-mz4cs\") pod \"maas-api-revoke-keys-e2e-api-b-804072-plz9c\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.674199 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.674115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:48.955720 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:48.955691 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c"] Sep 16 13:44:48.957873 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:44:48.957850 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e258c7_59db_4802_a1cf_e301ece299d9.slice/crio-a45ce862804c3abc452d232e28a419d874b41ea66a0cef6bc153ce3b94b98114 WatchSource:0}: Error finding container a45ce862804c3abc452d232e28a419d874b41ea66a0cef6bc153ce3b94b98114: Status 404 returned error can't find the container with id a45ce862804c3abc452d232e28a419d874b41ea66a0cef6bc153ce3b94b98114 Sep 16 13:44:49.029747 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:49.029716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" event={"ID":"d2e258c7-59db-4802-a1cf-e301ece299d9","Type":"ContainerStarted","Data":"a45ce862804c3abc452d232e28a419d874b41ea66a0cef6bc153ce3b94b98114"} Sep 16 13:44:49.504780 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:49.504685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:50.037923 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:50.037881 2581 generic.go:356] "Generic (PLEG): container finished" podID="d2e258c7-59db-4802-a1cf-e301ece299d9" containerID="979a94027d499a671c48f23fb68aca4f872d83c184890313c42358812eece692" exitCode=0 Sep 16 13:44:50.038349 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:50.037976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" event={"ID":"d2e258c7-59db-4802-a1cf-e301ece299d9","Type":"ContainerDied","Data":"979a94027d499a671c48f23fb68aca4f872d83c184890313c42358812eece692"} Sep 16 13:44:50.324067 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:50.324035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.105974 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.105932 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:51.186294 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.186259 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:51.294799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.294771 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz4cs\" (UniqueName: \"kubernetes.io/projected/d2e258c7-59db-4802-a1cf-e301ece299d9-kube-api-access-mz4cs\") pod \"d2e258c7-59db-4802-a1cf-e301ece299d9\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " Sep 16 13:44:51.295006 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.294808 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e258c7-59db-4802-a1cf-e301ece299d9-maas-api-service-ca\") pod \"d2e258c7-59db-4802-a1cf-e301ece299d9\" (UID: \"d2e258c7-59db-4802-a1cf-e301ece299d9\") " Sep 16 13:44:51.295260 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.295229 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e258c7-59db-4802-a1cf-e301ece299d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2e258c7-59db-4802-a1cf-e301ece299d9" (UID: "d2e258c7-59db-4802-a1cf-e301ece299d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:51.296900 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.296874 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e258c7-59db-4802-a1cf-e301ece299d9-kube-api-access-mz4cs" (OuterVolumeSpecName: "kube-api-access-mz4cs") pod "d2e258c7-59db-4802-a1cf-e301ece299d9" (UID: "d2e258c7-59db-4802-a1cf-e301ece299d9"). InnerVolumeSpecName "kube-api-access-mz4cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:51.395494 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.395469 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4cs\" (UniqueName: \"kubernetes.io/projected/d2e258c7-59db-4802-a1cf-e301ece299d9-kube-api-access-mz4cs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:51.395588 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:51.395501 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e258c7-59db-4802-a1cf-e301ece299d9-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:44:52.051352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:52.051319 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" Sep 16 13:44:52.051352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:52.051329 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c" event={"ID":"d2e258c7-59db-4802-a1cf-e301ece299d9","Type":"ContainerDied","Data":"a45ce862804c3abc452d232e28a419d874b41ea66a0cef6bc153ce3b94b98114"} Sep 16 13:44:52.051560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:52.051381 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45ce862804c3abc452d232e28a419d874b41ea66a0cef6bc153ce3b94b98114" Sep 16 13:44:52.155611 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:52.155567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:53.892381 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:53.892345 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:44:53.991483 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:44:53.991452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:00.161934 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.161901 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl"] Sep 16 13:45:00.162440 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.162337 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e258c7-59db-4802-a1cf-e301ece299d9" containerName="revoke-keys" Sep 16 13:45:00.162440 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.162349 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e258c7-59db-4802-a1cf-e301ece299d9" containerName="revoke-keys" Sep 16 13:45:00.162440 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.162426 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e258c7-59db-4802-a1cf-e301ece299d9" containerName="revoke-keys" Sep 16 13:45:00.165847 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.165830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:00.169526 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.169504 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt"] Sep 16 13:45:00.170510 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.170491 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:45:00.170624 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.170567 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:45:00.170908 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.170882 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:45:00.173025 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.173008 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms"] Sep 16 13:45:00.173141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.173123 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:00.176776 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.176760 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm"] Sep 16 13:45:00.176898 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.176881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:00.180049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.180035 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8"] Sep 16 13:45:00.180174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.180155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:00.183574 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.183552 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:00.197726 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.197702 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl"] Sep 16 13:45:00.202455 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.202431 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms"] Sep 16 13:45:00.206036 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.206014 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8"] Sep 16 13:45:00.210836 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.210817 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm"] Sep 16 13:45:00.213764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.213745 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt"] Sep 16 13:45:00.216374 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.216353 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc"] Sep 16 13:45:00.220425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.220402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:00.225257 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.225239 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v"] Sep 16 13:45:00.228819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.228800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:00.251162 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.251124 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc"] Sep 16 13:45:00.254664 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.254638 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v"] Sep 16 13:45:00.276865 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.276833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2dk\" (UniqueName: \"kubernetes.io/projected/2437b5eb-2e02-4764-991b-d5561d58f4b2-kube-api-access-6v2dk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt\" (UID: \"2437b5eb-2e02-4764-991b-d5561d58f4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:00.277000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.276878 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npfw\" (UniqueName: \"kubernetes.io/projected/8a377ef0-11c7-4c79-b929-e81de1893e36-kube-api-access-4npfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms\" (UID: \"8a377ef0-11c7-4c79-b929-e81de1893e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:00.277000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.276902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltqc\" (UniqueName: \"kubernetes.io/projected/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8-kube-api-access-qltqc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8\" (UID: \"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:00.277000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.276949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm2f\" (UniqueName: \"kubernetes.io/projected/56fe3e98-996d-4bbd-a89e-1560d9372eea-kube-api-access-6gm2f\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl\" (UID: \"56fe3e98-996d-4bbd-a89e-1560d9372eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:00.277000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.276988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztg8v\" (UniqueName: \"kubernetes.io/projected/3e38a71d-2a73-4aae-9d4f-04509f1f627f-kube-api-access-ztg8v\") pod \"maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm\" (UID: \"3e38a71d-2a73-4aae-9d4f-04509f1f627f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:00.377960 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.377923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztg8v\" (UniqueName: \"kubernetes.io/projected/3e38a71d-2a73-4aae-9d4f-04509f1f627f-kube-api-access-ztg8v\") pod \"maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm\" (UID: \"3e38a71d-2a73-4aae-9d4f-04509f1f627f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:00.378152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.377997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt69m\" (UniqueName: \"kubernetes.io/projected/412fa616-75fa-488c-a971-b481a6e429b7-kube-api-access-rt69m\") pod \"maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc\" (UID: \"412fa616-75fa-488c-a971-b481a6e429b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:00.378152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.378049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2dk\" (UniqueName: \"kubernetes.io/projected/2437b5eb-2e02-4764-991b-d5561d58f4b2-kube-api-access-6v2dk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt\" (UID: \"2437b5eb-2e02-4764-991b-d5561d58f4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:00.378152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.378129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npfw\" (UniqueName: \"kubernetes.io/projected/8a377ef0-11c7-4c79-b929-e81de1893e36-kube-api-access-4npfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms\" (UID: \"8a377ef0-11c7-4c79-b929-e81de1893e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:00.378356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.378179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv62z\" (UniqueName: \"kubernetes.io/projected/f0220603-ee86-41ed-8b34-b18887c8b211-kube-api-access-fv62z\") pod \"maas-api-key-cleanup-29826105-jhn6v\" (UID: \"f0220603-ee86-41ed-8b34-b18887c8b211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:00.378356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.378208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltqc\" (UniqueName: \"kubernetes.io/projected/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8-kube-api-access-qltqc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8\" (UID: \"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:00.378356 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.378269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm2f\" (UniqueName: \"kubernetes.io/projected/56fe3e98-996d-4bbd-a89e-1560d9372eea-kube-api-access-6gm2f\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl\" (UID: \"56fe3e98-996d-4bbd-a89e-1560d9372eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:00.386544 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.386516 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm2f\" (UniqueName: \"kubernetes.io/projected/56fe3e98-996d-4bbd-a89e-1560d9372eea-kube-api-access-6gm2f\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl\" (UID: \"56fe3e98-996d-4bbd-a89e-1560d9372eea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:00.386669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.386588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztg8v\" (UniqueName: \"kubernetes.io/projected/3e38a71d-2a73-4aae-9d4f-04509f1f627f-kube-api-access-ztg8v\") pod \"maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm\" (UID: \"3e38a71d-2a73-4aae-9d4f-04509f1f627f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:00.386843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.386822 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltqc\" (UniqueName: \"kubernetes.io/projected/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8-kube-api-access-qltqc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8\" (UID: \"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:00.387003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.386983 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npfw\" (UniqueName: \"kubernetes.io/projected/8a377ef0-11c7-4c79-b929-e81de1893e36-kube-api-access-4npfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms\" (UID: \"8a377ef0-11c7-4c79-b929-e81de1893e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:00.387101 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.387084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2dk\" (UniqueName: \"kubernetes.io/projected/2437b5eb-2e02-4764-991b-d5561d58f4b2-kube-api-access-6v2dk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt\" (UID: \"2437b5eb-2e02-4764-991b-d5561d58f4b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:00.477764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.477738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:00.479091 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.479068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv62z\" (UniqueName: \"kubernetes.io/projected/f0220603-ee86-41ed-8b34-b18887c8b211-kube-api-access-fv62z\") pod \"maas-api-key-cleanup-29826105-jhn6v\" (UID: \"f0220603-ee86-41ed-8b34-b18887c8b211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:00.479166 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.479152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt69m\" (UniqueName: \"kubernetes.io/projected/412fa616-75fa-488c-a971-b481a6e429b7-kube-api-access-rt69m\") pod \"maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc\" (UID: \"412fa616-75fa-488c-a971-b481a6e429b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:00.485696 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.485675 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:00.487021 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.486992 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv62z\" (UniqueName: \"kubernetes.io/projected/f0220603-ee86-41ed-8b34-b18887c8b211-kube-api-access-fv62z\") pod \"maas-api-key-cleanup-29826105-jhn6v\" (UID: \"f0220603-ee86-41ed-8b34-b18887c8b211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:00.487671 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.487648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt69m\" (UniqueName: \"kubernetes.io/projected/412fa616-75fa-488c-a971-b481a6e429b7-kube-api-access-rt69m\") pod \"maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc\" (UID: \"412fa616-75fa-488c-a971-b481a6e429b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:00.494625 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.494236 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:00.504541 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.504521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:00.514693 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.514654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:00.532920 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.532511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:00.540315 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.540194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:00.780589 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.780564 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl"] Sep 16 13:45:00.780995 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.780946 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fe3e98_996d_4bbd_a89e_1560d9372eea.slice/crio-5ea02eef7bec2d17ab3e503a99ce87c670eaf2f54c2fff221496111aafe404e1 WatchSource:0}: Error finding container 5ea02eef7bec2d17ab3e503a99ce87c670eaf2f54c2fff221496111aafe404e1: Status 404 returned error can't find the container with id 5ea02eef7bec2d17ab3e503a99ce87c670eaf2f54c2fff221496111aafe404e1 Sep 16 13:45:00.783182 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.783157 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2437b5eb_2e02_4764_991b_d5561d58f4b2.slice/crio-86ce4f7569b01a12d6871f8211446e78c358a0374c63d0348c4a70c4ca89b4f1 WatchSource:0}: Error finding container 86ce4f7569b01a12d6871f8211446e78c358a0374c63d0348c4a70c4ca89b4f1: Status 404 returned error can't find the container with id 86ce4f7569b01a12d6871f8211446e78c358a0374c63d0348c4a70c4ca89b4f1 Sep 16 13:45:00.785075 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.785048 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt"] Sep 16 13:45:00.797368 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.797342 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a377ef0_11c7_4c79_b929_e81de1893e36.slice/crio-5f7576559070217bf7942b8ba46c24a5fe858dbdabc5531ee480f5331cfa5059 WatchSource:0}: Error finding container 5f7576559070217bf7942b8ba46c24a5fe858dbdabc5531ee480f5331cfa5059: Status 404 returned error can't find the container with id 5f7576559070217bf7942b8ba46c24a5fe858dbdabc5531ee480f5331cfa5059 Sep 16 13:45:00.797783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.797758 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms"] Sep 16 13:45:00.856485 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.856454 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm"] Sep 16 13:45:00.857663 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.857635 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed7548e_6771_46e8_b3ba_e35a4e81f8d8.slice/crio-67a63ada3caa66bee7e9c579a2a974bf76c03454ca707cf8874b4483efbfb649 WatchSource:0}: Error finding container 67a63ada3caa66bee7e9c579a2a974bf76c03454ca707cf8874b4483efbfb649: Status 404 returned error can't find the container with id 67a63ada3caa66bee7e9c579a2a974bf76c03454ca707cf8874b4483efbfb649 Sep 16 13:45:00.858541 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.858518 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e38a71d_2a73_4aae_9d4f_04509f1f627f.slice/crio-5f9c0cc092bfd1b002d4df3c65da00f628c060435e339bea1bdc744183ef9a0d WatchSource:0}: Error finding container 5f9c0cc092bfd1b002d4df3c65da00f628c060435e339bea1bdc744183ef9a0d: Status 404 returned error can't find the container with id 5f9c0cc092bfd1b002d4df3c65da00f628c060435e339bea1bdc744183ef9a0d Sep 16 13:45:00.860417 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.860367 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8"] Sep 16 13:45:00.897052 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.897029 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v"] Sep 16 13:45:00.902471 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:00.902446 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc"] Sep 16 13:45:00.987858 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.987824 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0220603_ee86_41ed_8b34_b18887c8b211.slice/crio-a5f87f470fde5880c6496e87e436c59f95920e13c3bc010fcd19351e30c75d5a WatchSource:0}: Error finding container a5f87f470fde5880c6496e87e436c59f95920e13c3bc010fcd19351e30c75d5a: Status 404 returned error can't find the container with id a5f87f470fde5880c6496e87e436c59f95920e13c3bc010fcd19351e30c75d5a Sep 16 13:45:00.988154 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:00.988132 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412fa616_75fa_488c_a971_b481a6e429b7.slice/crio-8de8f76bbaca1f751f89a7ebe7a5b3a53f04c24ffeab31933603269687588bd8 WatchSource:0}: Error finding container 8de8f76bbaca1f751f89a7ebe7a5b3a53f04c24ffeab31933603269687588bd8: Status 404 returned error can't find the container with id 8de8f76bbaca1f751f89a7ebe7a5b3a53f04c24ffeab31933603269687588bd8 Sep 16 13:45:01.111435 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.111407 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" event={"ID":"3e38a71d-2a73-4aae-9d4f-04509f1f627f","Type":"ContainerStarted","Data":"5f9c0cc092bfd1b002d4df3c65da00f628c060435e339bea1bdc744183ef9a0d"} Sep 16 13:45:01.112651 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.112622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" event={"ID":"412fa616-75fa-488c-a971-b481a6e429b7","Type":"ContainerStarted","Data":"8de8f76bbaca1f751f89a7ebe7a5b3a53f04c24ffeab31933603269687588bd8"} Sep 16 13:45:01.113799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.113778 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" event={"ID":"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8","Type":"ContainerStarted","Data":"67a63ada3caa66bee7e9c579a2a974bf76c03454ca707cf8874b4483efbfb649"} Sep 16 13:45:01.114893 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.114868 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" event={"ID":"2437b5eb-2e02-4764-991b-d5561d58f4b2","Type":"ContainerStarted","Data":"86ce4f7569b01a12d6871f8211446e78c358a0374c63d0348c4a70c4ca89b4f1"} Sep 16 13:45:01.115897 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.115874 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" event={"ID":"f0220603-ee86-41ed-8b34-b18887c8b211","Type":"ContainerStarted","Data":"a5f87f470fde5880c6496e87e436c59f95920e13c3bc010fcd19351e30c75d5a"} Sep 16 13:45:01.117030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.117011 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" event={"ID":"56fe3e98-996d-4bbd-a89e-1560d9372eea","Type":"ContainerStarted","Data":"5ea02eef7bec2d17ab3e503a99ce87c670eaf2f54c2fff221496111aafe404e1"} Sep 16 13:45:01.118093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:01.118073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" event={"ID":"8a377ef0-11c7-4c79-b929-e81de1893e36","Type":"ContainerStarted","Data":"5f7576559070217bf7942b8ba46c24a5fe858dbdabc5531ee480f5331cfa5059"} Sep 16 13:45:02.128070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.128033 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" event={"ID":"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8","Type":"ContainerStarted","Data":"3ac148e2b4c4dda54e6774ee13acd0ac4c9709145cdebeaba32500fcc9a6b308"} Sep 16 13:45:02.129647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.129620 2581 generic.go:356] "Generic (PLEG): container finished" podID="2437b5eb-2e02-4764-991b-d5561d58f4b2" containerID="7d1cae88a350864e39d183f296ae4ef2359e01bbed59cbd3e3571fd35ef1c228" exitCode=0 Sep 16 13:45:02.129744 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.129697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" event={"ID":"2437b5eb-2e02-4764-991b-d5561d58f4b2","Type":"ContainerDied","Data":"7d1cae88a350864e39d183f296ae4ef2359e01bbed59cbd3e3571fd35ef1c228"} Sep 16 13:45:02.131256 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.131235 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" event={"ID":"f0220603-ee86-41ed-8b34-b18887c8b211","Type":"ContainerStarted","Data":"221504f95eb6561b42ee97860159e8af5b2e497ec354232e93c50624cb2266ad"} Sep 16 13:45:02.132837 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.132819 2581 generic.go:356] "Generic (PLEG): container finished" podID="56fe3e98-996d-4bbd-a89e-1560d9372eea" containerID="31599e0514d60a05231b0561456ba5a4a2dbc0bc577e9ac3bd1d96a2ffebc143" exitCode=0 Sep 16 13:45:02.132928 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.132845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" event={"ID":"56fe3e98-996d-4bbd-a89e-1560d9372eea","Type":"ContainerDied","Data":"31599e0514d60a05231b0561456ba5a4a2dbc0bc577e9ac3bd1d96a2ffebc143"} Sep 16 13:45:02.134375 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.134355 2581 generic.go:356] "Generic (PLEG): container finished" podID="8a377ef0-11c7-4c79-b929-e81de1893e36" containerID="a1d9798364009ddbef9e2dc2fb13f3f115ef06cfd8ba07ceafbb965dbfdb8132" exitCode=0 Sep 16 13:45:02.134475 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.134421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" event={"ID":"8a377ef0-11c7-4c79-b929-e81de1893e36","Type":"ContainerDied","Data":"a1d9798364009ddbef9e2dc2fb13f3f115ef06cfd8ba07ceafbb965dbfdb8132"} Sep 16 13:45:02.135905 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.135886 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" event={"ID":"3e38a71d-2a73-4aae-9d4f-04509f1f627f","Type":"ContainerStarted","Data":"ed998fb03de2e1951a62fa53e3d3fb8d5ab017fb613aa5535e2d67caba66b910"} Sep 16 13:45:02.137376 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.137357 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" event={"ID":"412fa616-75fa-488c-a971-b481a6e429b7","Type":"ContainerStarted","Data":"5b9249efeba6098388e9bb29f1faf5089e2375701ed4784630c4fa67c4e18648"} Sep 16 13:45:02.149000 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.148963 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" podStartSLOduration=2.148949654 podStartE2EDuration="2.148949654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:02.147904664 +0000 UTC m=+2187.267631285" watchObservedRunningTime="2026-09-16 13:45:02.148949654 +0000 UTC m=+2187.268676193" Sep 16 13:45:02.282255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.282150 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" podStartSLOduration=2.28213566 podStartE2EDuration="2.28213566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:02.279703207 +0000 UTC m=+2187.399429849" watchObservedRunningTime="2026-09-16 13:45:02.28213566 +0000 UTC m=+2187.401862208" Sep 16 13:45:02.317403 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.317354 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" podStartSLOduration=2.317340186 podStartE2EDuration="2.317340186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:02.310727319 +0000 UTC m=+2187.430453893" watchObservedRunningTime="2026-09-16 13:45:02.317340186 +0000 UTC m=+2187.437066734" Sep 16 13:45:02.340958 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:02.340914 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" podStartSLOduration=2.340898589 podStartE2EDuration="2.340898589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:02.338277773 +0000 UTC m=+2187.458004321" watchObservedRunningTime="2026-09-16 13:45:02.340898589 +0000 UTC m=+2187.460625137" Sep 16 13:45:03.154354 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.154305 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0220603-ee86-41ed-8b34-b18887c8b211" containerID="221504f95eb6561b42ee97860159e8af5b2e497ec354232e93c50624cb2266ad" exitCode=0 Sep 16 13:45:03.154774 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.154396 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" event={"ID":"f0220603-ee86-41ed-8b34-b18887c8b211","Type":"ContainerDied","Data":"221504f95eb6561b42ee97860159e8af5b2e497ec354232e93c50624cb2266ad"} Sep 16 13:45:03.156024 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.155996 2581 generic.go:356] "Generic (PLEG): container finished" podID="3e38a71d-2a73-4aae-9d4f-04509f1f627f" containerID="ed998fb03de2e1951a62fa53e3d3fb8d5ab017fb613aa5535e2d67caba66b910" exitCode=0 Sep 16 13:45:03.156165 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.156028 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" event={"ID":"3e38a71d-2a73-4aae-9d4f-04509f1f627f","Type":"ContainerDied","Data":"ed998fb03de2e1951a62fa53e3d3fb8d5ab017fb613aa5535e2d67caba66b910"} Sep 16 13:45:03.157509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.157490 2581 generic.go:356] "Generic (PLEG): container finished" podID="412fa616-75fa-488c-a971-b481a6e429b7" containerID="5b9249efeba6098388e9bb29f1faf5089e2375701ed4784630c4fa67c4e18648" exitCode=0 Sep 16 13:45:03.157598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.157516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" event={"ID":"412fa616-75fa-488c-a971-b481a6e429b7","Type":"ContainerDied","Data":"5b9249efeba6098388e9bb29f1faf5089e2375701ed4784630c4fa67c4e18648"} Sep 16 13:45:03.159001 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.158980 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ed7548e-6771-46e8-b3ba-e35a4e81f8d8" containerID="3ac148e2b4c4dda54e6774ee13acd0ac4c9709145cdebeaba32500fcc9a6b308" exitCode=0 Sep 16 13:45:03.159099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.159079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" event={"ID":"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8","Type":"ContainerDied","Data":"3ac148e2b4c4dda54e6774ee13acd0ac4c9709145cdebeaba32500fcc9a6b308"} Sep 16 13:45:03.307343 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.307313 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:03.350346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.350264 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:03.375542 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.375513 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:03.410695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.410622 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gm2f\" (UniqueName: \"kubernetes.io/projected/56fe3e98-996d-4bbd-a89e-1560d9372eea-kube-api-access-6gm2f\") pod \"56fe3e98-996d-4bbd-a89e-1560d9372eea\" (UID: \"56fe3e98-996d-4bbd-a89e-1560d9372eea\") " Sep 16 13:45:03.412681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.412651 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fe3e98-996d-4bbd-a89e-1560d9372eea-kube-api-access-6gm2f" (OuterVolumeSpecName: "kube-api-access-6gm2f") pod "56fe3e98-996d-4bbd-a89e-1560d9372eea" (UID: "56fe3e98-996d-4bbd-a89e-1560d9372eea"). InnerVolumeSpecName "kube-api-access-6gm2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:03.511458 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.511425 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4npfw\" (UniqueName: \"kubernetes.io/projected/8a377ef0-11c7-4c79-b929-e81de1893e36-kube-api-access-4npfw\") pod \"8a377ef0-11c7-4c79-b929-e81de1893e36\" (UID: \"8a377ef0-11c7-4c79-b929-e81de1893e36\") " Sep 16 13:45:03.511624 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.511492 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v2dk\" (UniqueName: \"kubernetes.io/projected/2437b5eb-2e02-4764-991b-d5561d58f4b2-kube-api-access-6v2dk\") pod \"2437b5eb-2e02-4764-991b-d5561d58f4b2\" (UID: \"2437b5eb-2e02-4764-991b-d5561d58f4b2\") " Sep 16 13:45:03.511855 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.511826 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm2f\" (UniqueName: \"kubernetes.io/projected/56fe3e98-996d-4bbd-a89e-1560d9372eea-kube-api-access-6gm2f\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:03.513617 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.513592 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2437b5eb-2e02-4764-991b-d5561d58f4b2-kube-api-access-6v2dk" (OuterVolumeSpecName: "kube-api-access-6v2dk") pod "2437b5eb-2e02-4764-991b-d5561d58f4b2" (UID: "2437b5eb-2e02-4764-991b-d5561d58f4b2"). InnerVolumeSpecName "kube-api-access-6v2dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:03.513617 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.513603 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a377ef0-11c7-4c79-b929-e81de1893e36-kube-api-access-4npfw" (OuterVolumeSpecName: "kube-api-access-4npfw") pod "8a377ef0-11c7-4c79-b929-e81de1893e36" (UID: "8a377ef0-11c7-4c79-b929-e81de1893e36"). InnerVolumeSpecName "kube-api-access-4npfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:03.612367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.612333 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npfw\" (UniqueName: \"kubernetes.io/projected/8a377ef0-11c7-4c79-b929-e81de1893e36-kube-api-access-4npfw\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:03.612367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:03.612361 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v2dk\" (UniqueName: \"kubernetes.io/projected/2437b5eb-2e02-4764-991b-d5561d58f4b2-kube-api-access-6v2dk\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.066650 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.066611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:04.122247 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.122217 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8"] Sep 16 13:45:04.122461 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.122429 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" podUID="0462f3b3-2002-4991-967d-6028145dfdb6" containerName="payload-processing" containerID="cri-o://45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d" gracePeriod=30 Sep 16 13:45:04.137222 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.137191 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx"] Sep 16 13:45:04.137518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.137487 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" podUID="6bef7c7d-033d-4383-8685-aa5c9ceb979d" containerName="payload-pre-processing" containerID="cri-o://09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0" gracePeriod=30 Sep 16 13:45:04.168554 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.168531 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" Sep 16 13:45:04.169049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.168581 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt" event={"ID":"2437b5eb-2e02-4764-991b-d5561d58f4b2","Type":"ContainerDied","Data":"86ce4f7569b01a12d6871f8211446e78c358a0374c63d0348c4a70c4ca89b4f1"} Sep 16 13:45:04.169049 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.168633 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ce4f7569b01a12d6871f8211446e78c358a0374c63d0348c4a70c4ca89b4f1" Sep 16 13:45:04.170205 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.170182 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" Sep 16 13:45:04.170357 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.170182 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl" event={"ID":"56fe3e98-996d-4bbd-a89e-1560d9372eea","Type":"ContainerDied","Data":"5ea02eef7bec2d17ab3e503a99ce87c670eaf2f54c2fff221496111aafe404e1"} Sep 16 13:45:04.170412 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.170360 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea02eef7bec2d17ab3e503a99ce87c670eaf2f54c2fff221496111aafe404e1" Sep 16 13:45:04.171787 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.171757 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" Sep 16 13:45:04.171911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.171801 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms" event={"ID":"8a377ef0-11c7-4c79-b929-e81de1893e36","Type":"ContainerDied","Data":"5f7576559070217bf7942b8ba46c24a5fe858dbdabc5531ee480f5331cfa5059"} Sep 16 13:45:04.171911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.171842 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7576559070217bf7942b8ba46c24a5fe858dbdabc5531ee480f5331cfa5059" Sep 16 13:45:04.178354 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.178257 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:04.271829 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.271800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:04.357711 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.357648 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c"] Sep 16 13:45:04.366094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.366025 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-804072-plz9c"] Sep 16 13:45:04.370705 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.370646 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:04.396099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.393894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:04.506626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.506598 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:04.509535 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.509513 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:04.513367 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.513348 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:04.516666 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.516650 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:45:04.520117 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.520100 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:45:04.524795 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.524761 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv62z\" (UniqueName: \"kubernetes.io/projected/f0220603-ee86-41ed-8b34-b18887c8b211-kube-api-access-fv62z\") pod \"f0220603-ee86-41ed-8b34-b18887c8b211\" (UID: \"f0220603-ee86-41ed-8b34-b18887c8b211\") " Sep 16 13:45:04.527295 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.527256 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0220603-ee86-41ed-8b34-b18887c8b211-kube-api-access-fv62z" (OuterVolumeSpecName: "kube-api-access-fv62z") pod "f0220603-ee86-41ed-8b34-b18887c8b211" (UID: "f0220603-ee86-41ed-8b34-b18887c8b211"). InnerVolumeSpecName "kube-api-access-fv62z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:04.625815 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625730 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltqc\" (UniqueName: \"kubernetes.io/projected/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8-kube-api-access-qltqc\") pod \"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8\" (UID: \"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8\") " Sep 16 13:45:04.625815 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625797 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztg8v\" (UniqueName: \"kubernetes.io/projected/3e38a71d-2a73-4aae-9d4f-04509f1f627f-kube-api-access-ztg8v\") pod \"3e38a71d-2a73-4aae-9d4f-04509f1f627f\" (UID: \"3e38a71d-2a73-4aae-9d4f-04509f1f627f\") " Sep 16 13:45:04.626099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625825 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0462f3b3-2002-4991-967d-6028145dfdb6-plugins-config-volume\") pod \"0462f3b3-2002-4991-967d-6028145dfdb6\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " Sep 16 13:45:04.626099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625848 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt69m\" (UniqueName: \"kubernetes.io/projected/412fa616-75fa-488c-a971-b481a6e429b7-kube-api-access-rt69m\") pod \"412fa616-75fa-488c-a971-b481a6e429b7\" (UID: \"412fa616-75fa-488c-a971-b481a6e429b7\") " Sep 16 13:45:04.626099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625903 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bef7c7d-033d-4383-8685-aa5c9ceb979d-plugins-config-volume\") pod \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " Sep 16 13:45:04.626099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625966 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbrtm\" (UniqueName: \"kubernetes.io/projected/6bef7c7d-033d-4383-8685-aa5c9ceb979d-kube-api-access-dbrtm\") pod \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\" (UID: \"6bef7c7d-033d-4383-8685-aa5c9ceb979d\") " Sep 16 13:45:04.626099 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.625995 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x64rc\" (UniqueName: \"kubernetes.io/projected/0462f3b3-2002-4991-967d-6028145dfdb6-kube-api-access-x64rc\") pod \"0462f3b3-2002-4991-967d-6028145dfdb6\" (UID: \"0462f3b3-2002-4991-967d-6028145dfdb6\") " Sep 16 13:45:04.626404 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.626222 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0462f3b3-2002-4991-967d-6028145dfdb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0462f3b3-2002-4991-967d-6028145dfdb6" (UID: "0462f3b3-2002-4991-967d-6028145dfdb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:04.626404 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.626321 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bef7c7d-033d-4383-8685-aa5c9ceb979d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bef7c7d-033d-4383-8685-aa5c9ceb979d" (UID: "6bef7c7d-033d-4383-8685-aa5c9ceb979d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:04.626702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.626650 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bef7c7d-033d-4383-8685-aa5c9ceb979d-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.626702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.626678 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv62z\" (UniqueName: \"kubernetes.io/projected/f0220603-ee86-41ed-8b34-b18887c8b211-kube-api-access-fv62z\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.626702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.626692 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0462f3b3-2002-4991-967d-6028145dfdb6-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.628093 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.628076 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412fa616-75fa-488c-a971-b481a6e429b7-kube-api-access-rt69m" (OuterVolumeSpecName: "kube-api-access-rt69m") pod "412fa616-75fa-488c-a971-b481a6e429b7" (UID: "412fa616-75fa-488c-a971-b481a6e429b7"). InnerVolumeSpecName "kube-api-access-rt69m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:04.628437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.628416 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8-kube-api-access-qltqc" (OuterVolumeSpecName: "kube-api-access-qltqc") pod "1ed7548e-6771-46e8-b3ba-e35a4e81f8d8" (UID: "1ed7548e-6771-46e8-b3ba-e35a4e81f8d8"). InnerVolumeSpecName "kube-api-access-qltqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:04.628522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.628478 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bef7c7d-033d-4383-8685-aa5c9ceb979d-kube-api-access-dbrtm" (OuterVolumeSpecName: "kube-api-access-dbrtm") pod "6bef7c7d-033d-4383-8685-aa5c9ceb979d" (UID: "6bef7c7d-033d-4383-8685-aa5c9ceb979d"). InnerVolumeSpecName "kube-api-access-dbrtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:04.628522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.628510 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e38a71d-2a73-4aae-9d4f-04509f1f627f-kube-api-access-ztg8v" (OuterVolumeSpecName: "kube-api-access-ztg8v") pod "3e38a71d-2a73-4aae-9d4f-04509f1f627f" (UID: "3e38a71d-2a73-4aae-9d4f-04509f1f627f"). InnerVolumeSpecName "kube-api-access-ztg8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:04.628608 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.628531 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0462f3b3-2002-4991-967d-6028145dfdb6-kube-api-access-x64rc" (OuterVolumeSpecName: "kube-api-access-x64rc") pod "0462f3b3-2002-4991-967d-6028145dfdb6" (UID: "0462f3b3-2002-4991-967d-6028145dfdb6"). InnerVolumeSpecName "kube-api-access-x64rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:04.727265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.727218 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztg8v\" (UniqueName: \"kubernetes.io/projected/3e38a71d-2a73-4aae-9d4f-04509f1f627f-kube-api-access-ztg8v\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.727265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.727260 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt69m\" (UniqueName: \"kubernetes.io/projected/412fa616-75fa-488c-a971-b481a6e429b7-kube-api-access-rt69m\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.727265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.727270 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrtm\" (UniqueName: \"kubernetes.io/projected/6bef7c7d-033d-4383-8685-aa5c9ceb979d-kube-api-access-dbrtm\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.727265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.727302 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64rc\" (UniqueName: \"kubernetes.io/projected/0462f3b3-2002-4991-967d-6028145dfdb6-kube-api-access-x64rc\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.727265 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.727312 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltqc\" (UniqueName: \"kubernetes.io/projected/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8-kube-api-access-qltqc\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:04.758119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.758084 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:04.859346 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:04.859316 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:05.179785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.179749 2581 generic.go:356] "Generic (PLEG): container finished" podID="6bef7c7d-033d-4383-8685-aa5c9ceb979d" containerID="09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0" exitCode=0 Sep 16 13:45:05.180226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.179790 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" Sep 16 13:45:05.180226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.179791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" event={"ID":"6bef7c7d-033d-4383-8685-aa5c9ceb979d","Type":"ContainerDied","Data":"09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0"} Sep 16 13:45:05.180226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.179843 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx" event={"ID":"6bef7c7d-033d-4383-8685-aa5c9ceb979d","Type":"ContainerDied","Data":"edc50b02b22476ffb9941f69bea14f28710bb186e02235fbc89488a331b39bd9"} Sep 16 13:45:05.180226 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.179870 2581 scope.go:117] "RemoveContainer" containerID="09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0" Sep 16 13:45:05.181338 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.181316 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" Sep 16 13:45:05.181482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.181317 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826105-jhn6v" event={"ID":"f0220603-ee86-41ed-8b34-b18887c8b211","Type":"ContainerDied","Data":"a5f87f470fde5880c6496e87e436c59f95920e13c3bc010fcd19351e30c75d5a"} Sep 16 13:45:05.181482 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.181458 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f87f470fde5880c6496e87e436c59f95920e13c3bc010fcd19351e30c75d5a" Sep 16 13:45:05.182998 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.182951 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" event={"ID":"3e38a71d-2a73-4aae-9d4f-04509f1f627f","Type":"ContainerDied","Data":"5f9c0cc092bfd1b002d4df3c65da00f628c060435e339bea1bdc744183ef9a0d"} Sep 16 13:45:05.183134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.183006 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f9c0cc092bfd1b002d4df3c65da00f628c060435e339bea1bdc744183ef9a0d" Sep 16 13:45:05.183134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.182974 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm" Sep 16 13:45:05.184669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.184637 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" Sep 16 13:45:05.184669 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.184654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc" event={"ID":"412fa616-75fa-488c-a971-b481a6e429b7","Type":"ContainerDied","Data":"8de8f76bbaca1f751f89a7ebe7a5b3a53f04c24ffeab31933603269687588bd8"} Sep 16 13:45:05.184882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.184690 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de8f76bbaca1f751f89a7ebe7a5b3a53f04c24ffeab31933603269687588bd8" Sep 16 13:45:05.186772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.186740 2581 generic.go:356] "Generic (PLEG): container finished" podID="0462f3b3-2002-4991-967d-6028145dfdb6" containerID="45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d" exitCode=0 Sep 16 13:45:05.186888 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.186872 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" event={"ID":"0462f3b3-2002-4991-967d-6028145dfdb6","Type":"ContainerDied","Data":"45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d"} Sep 16 13:45:05.186959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.186912 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" event={"ID":"0462f3b3-2002-4991-967d-6028145dfdb6","Type":"ContainerDied","Data":"8025e6d8b5944a4d225b1405d094695c3666e173566136e7640f2fc1c38ed71a"} Sep 16 13:45:05.187015 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.186979 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8" Sep 16 13:45:05.188853 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.188830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" event={"ID":"1ed7548e-6771-46e8-b3ba-e35a4e81f8d8","Type":"ContainerDied","Data":"67a63ada3caa66bee7e9c579a2a974bf76c03454ca707cf8874b4483efbfb649"} Sep 16 13:45:05.188944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.188875 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a63ada3caa66bee7e9c579a2a974bf76c03454ca707cf8874b4483efbfb649" Sep 16 13:45:05.188944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.188884 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8" Sep 16 13:45:05.194269 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.194245 2581 scope.go:117] "RemoveContainer" containerID="09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0" Sep 16 13:45:05.198636 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:45:05.198429 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0\": container with ID starting with 09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0 not found: ID does not exist" containerID="09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0" Sep 16 13:45:05.199044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.198781 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0"} err="failed to get container status \"09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0\": rpc error: code = NotFound desc = could not find container \"09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0\": container with ID starting with 09564667568a6b89a6e7c710d460f1add31b6a4961d3e2c89ae060efe8be33f0 not found: ID does not exist" Sep 16 13:45:05.199173 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.199158 2581 scope.go:117] "RemoveContainer" containerID="45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d" Sep 16 13:45:05.215016 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.214858 2581 scope.go:117] "RemoveContainer" containerID="45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d" Sep 16 13:45:05.216016 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:45:05.215984 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d\": container with ID starting with 45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d not found: ID does not exist" containerID="45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d" Sep 16 13:45:05.216123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.216028 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d"} err="failed to get container status \"45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d\": rpc error: code = NotFound desc = could not find container \"45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d\": container with ID starting with 45d7838fc26b8a42c260efe04d311baa99e765d09e1f506d8904307c7c9b9a9d not found: ID does not exist" Sep 16 13:45:05.223522 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.223498 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx"] Sep 16 13:45:05.228920 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.228897 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-804072-657c5d5994-t2gbx"] Sep 16 13:45:05.251510 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.251478 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8"] Sep 16 13:45:05.256715 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.256695 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-804072-5b6788555f-hlvd8"] Sep 16 13:45:05.388704 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.388672 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0462f3b3-2002-4991-967d-6028145dfdb6" path="/var/lib/kubelet/pods/0462f3b3-2002-4991-967d-6028145dfdb6/volumes" Sep 16 13:45:05.389058 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.389044 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bef7c7d-033d-4383-8685-aa5c9ceb979d" path="/var/lib/kubelet/pods/6bef7c7d-033d-4383-8685-aa5c9ceb979d/volumes" Sep 16 13:45:05.389400 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.389386 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e258c7-59db-4802-a1cf-e301ece299d9" path="/var/lib/kubelet/pods/d2e258c7-59db-4802-a1cf-e301ece299d9/volumes" Sep 16 13:45:05.609481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:05.609448 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:06.540480 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:06.540446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:07.316005 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:07.315961 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:08.070624 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:08.070590 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:11.369846 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:11.369808 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:11.475492 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:11.475455 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:11.810942 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:11.810908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:12.122085 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122008 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd"] Sep 16 13:45:12.122462 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122449 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed7548e-6771-46e8-b3ba-e35a4e81f8d8" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122464 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed7548e-6771-46e8-b3ba-e35a4e81f8d8" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122476 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412fa616-75fa-488c-a971-b481a6e429b7" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122482 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="412fa616-75fa-488c-a971-b481a6e429b7" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122489 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bef7c7d-033d-4383-8685-aa5c9ceb979d" containerName="payload-pre-processing" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122494 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bef7c7d-033d-4383-8685-aa5c9ceb979d" containerName="payload-pre-processing" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122503 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56fe3e98-996d-4bbd-a89e-1560d9372eea" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122508 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56fe3e98-996d-4bbd-a89e-1560d9372eea" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122513 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2437b5eb-2e02-4764-991b-d5561d58f4b2" containerName="cleanup" Sep 16 13:45:12.122517 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122518 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2437b5eb-2e02-4764-991b-d5561d58f4b2" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122530 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0220603-ee86-41ed-8b34-b18887c8b211" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122538 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0220603-ee86-41ed-8b34-b18887c8b211" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122549 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0462f3b3-2002-4991-967d-6028145dfdb6" containerName="payload-processing" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122555 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0462f3b3-2002-4991-967d-6028145dfdb6" containerName="payload-processing" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122562 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a377ef0-11c7-4c79-b929-e81de1893e36" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122566 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a377ef0-11c7-4c79-b929-e81de1893e36" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122578 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e38a71d-2a73-4aae-9d4f-04509f1f627f" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122582 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e38a71d-2a73-4aae-9d4f-04509f1f627f" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122648 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bef7c7d-033d-4383-8685-aa5c9ceb979d" containerName="payload-pre-processing" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122656 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0462f3b3-2002-4991-967d-6028145dfdb6" containerName="payload-processing" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122661 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2437b5eb-2e02-4764-991b-d5561d58f4b2" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122666 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e38a71d-2a73-4aae-9d4f-04509f1f627f" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122673 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56fe3e98-996d-4bbd-a89e-1560d9372eea" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122678 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed7548e-6771-46e8-b3ba-e35a4e81f8d8" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122685 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a377ef0-11c7-4c79-b929-e81de1893e36" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122689 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="412fa616-75fa-488c-a971-b481a6e429b7" containerName="cleanup" Sep 16 13:45:12.122781 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.122694 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0220603-ee86-41ed-8b34-b18887c8b211" containerName="cleanup" Sep 16 13:45:12.125989 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.125972 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.129464 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.129442 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:45:12.129914 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.129894 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:45:12.130003 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.129919 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:45:12.142767 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.142740 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd"] Sep 16 13:45:12.200772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.200742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/aa1a710b-dc96-455e-bfd1-80c3b55128b0-kube-api-access-g4glb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.200921 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.200875 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a710b-dc96-455e-bfd1-80c3b55128b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.301626 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.301588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/aa1a710b-dc96-455e-bfd1-80c3b55128b0-kube-api-access-g4glb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.301784 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.301720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a710b-dc96-455e-bfd1-80c3b55128b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.302360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.302339 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a710b-dc96-455e-bfd1-80c3b55128b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.310394 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.310369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/aa1a710b-dc96-455e-bfd1-80c3b55128b0-kube-api-access-g4glb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.397548 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.397461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:12.436345 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.436319 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:12.717543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:12.717508 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd"] Sep 16 13:45:12.718843 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:12.718809 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1a710b_dc96_455e_bfd1_80c3b55128b0.slice/crio-2e86a17d3cfd0443979299fb4b378b15dbb49606ed9eb56c5623e735469763d3 WatchSource:0}: Error finding container 2e86a17d3cfd0443979299fb4b378b15dbb49606ed9eb56c5623e735469763d3: Status 404 returned error can't find the container with id 2e86a17d3cfd0443979299fb4b378b15dbb49606ed9eb56c5623e735469763d3 Sep 16 13:45:13.244333 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:13.244219 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" event={"ID":"aa1a710b-dc96-455e-bfd1-80c3b55128b0","Type":"ContainerStarted","Data":"3f02d24defc5c92da46c43409e0d091330d2e68a0e5e2e8e872428031d6c3869"} Sep 16 13:45:13.244333 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:13.244291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" event={"ID":"aa1a710b-dc96-455e-bfd1-80c3b55128b0","Type":"ContainerStarted","Data":"2e86a17d3cfd0443979299fb4b378b15dbb49606ed9eb56c5623e735469763d3"} Sep 16 13:45:13.246248 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:13.246221 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:13.261425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:13.261385 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" podStartSLOduration=1.261370493 podStartE2EDuration="1.261370493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:13.260248097 +0000 UTC m=+2198.379974666" watchObservedRunningTime="2026-09-16 13:45:13.261370493 +0000 UTC m=+2198.381097041" Sep 16 13:45:13.937442 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:13.937405 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:14.253157 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:14.253125 2581 generic.go:356] "Generic (PLEG): container finished" podID="aa1a710b-dc96-455e-bfd1-80c3b55128b0" containerID="3f02d24defc5c92da46c43409e0d091330d2e68a0e5e2e8e872428031d6c3869" exitCode=0 Sep 16 13:45:14.253358 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:14.253211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" event={"ID":"aa1a710b-dc96-455e-bfd1-80c3b55128b0","Type":"ContainerDied","Data":"3f02d24defc5c92da46c43409e0d091330d2e68a0e5e2e8e872428031d6c3869"} Sep 16 13:45:14.742612 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:14.742574 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:15.397397 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.397371 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:15.428518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.428490 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/aa1a710b-dc96-455e-bfd1-80c3b55128b0-kube-api-access-g4glb\") pod \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " Sep 16 13:45:15.428671 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.428592 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a710b-dc96-455e-bfd1-80c3b55128b0-maas-api-service-ca\") pod \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\" (UID: \"aa1a710b-dc96-455e-bfd1-80c3b55128b0\") " Sep 16 13:45:15.429212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.429183 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1a710b-dc96-455e-bfd1-80c3b55128b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa1a710b-dc96-455e-bfd1-80c3b55128b0" (UID: "aa1a710b-dc96-455e-bfd1-80c3b55128b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:15.431310 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.431289 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1a710b-dc96-455e-bfd1-80c3b55128b0-kube-api-access-g4glb" (OuterVolumeSpecName: "kube-api-access-g4glb") pod "aa1a710b-dc96-455e-bfd1-80c3b55128b0" (UID: "aa1a710b-dc96-455e-bfd1-80c3b55128b0"). InnerVolumeSpecName "kube-api-access-g4glb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:15.529516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.529432 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/aa1a710b-dc96-455e-bfd1-80c3b55128b0-kube-api-access-g4glb\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:15.529516 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.529468 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1a710b-dc96-455e-bfd1-80c3b55128b0-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:15.705054 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:15.705017 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:16.267218 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:16.267184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" event={"ID":"aa1a710b-dc96-455e-bfd1-80c3b55128b0","Type":"ContainerDied","Data":"2e86a17d3cfd0443979299fb4b378b15dbb49606ed9eb56c5623e735469763d3"} Sep 16 13:45:16.267218 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:16.267206 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd" Sep 16 13:45:16.267436 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:16.267233 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e86a17d3cfd0443979299fb4b378b15dbb49606ed9eb56c5623e735469763d3" Sep 16 13:45:16.598071 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:16.597980 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:22.769071 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:22.769034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:22.807957 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:22.807920 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz"] Sep 16 13:45:22.808228 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:22.808197 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" podUID="c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" containerName="payload-processing" containerID="cri-o://cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8" gracePeriod=30 Sep 16 13:45:22.881961 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:22.881926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:22.977432 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:22.977402 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:23.071788 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.071763 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:45:23.080500 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.080478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:23.097246 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.097211 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-plugins-config-volume\") pod \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " Sep 16 13:45:23.097399 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.097276 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52fm6\" (UniqueName: \"kubernetes.io/projected/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-kube-api-access-52fm6\") pod \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\" (UID: \"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23\") " Sep 16 13:45:23.097681 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.097645 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" (UID: "c9982d77-4da2-4a7b-8af8-a2bbfb8eab23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:23.100150 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.100123 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-kube-api-access-52fm6" (OuterVolumeSpecName: "kube-api-access-52fm6") pod "c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" (UID: "c9982d77-4da2-4a7b-8af8-a2bbfb8eab23"). InnerVolumeSpecName "kube-api-access-52fm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:23.107595 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.107570 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd"] Sep 16 13:45:23.116844 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.116819 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b0514e-xwmwd"] Sep 16 13:45:23.163170 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.163134 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:23.198762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.198725 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:23.198762 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.198757 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52fm6\" (UniqueName: \"kubernetes.io/projected/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23-kube-api-access-52fm6\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:23.313656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.313569 2581 generic.go:356] "Generic (PLEG): container finished" podID="c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" containerID="cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8" exitCode=0 Sep 16 13:45:23.313656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.313613 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" Sep 16 13:45:23.313868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.313660 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" event={"ID":"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23","Type":"ContainerDied","Data":"cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8"} Sep 16 13:45:23.313868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.313713 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz" event={"ID":"c9982d77-4da2-4a7b-8af8-a2bbfb8eab23","Type":"ContainerDied","Data":"a9cb08430046649151e3244f01e02acda719ae9489ddac4bf6c18803dd93d0b6"} Sep 16 13:45:23.313868 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.313736 2581 scope.go:117] "RemoveContainer" containerID="cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8" Sep 16 13:45:23.326415 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.326380 2581 scope.go:117] "RemoveContainer" containerID="cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8" Sep 16 13:45:23.326698 ip-10-0-133-73 kubenswrapper[2581]: E0916 13:45:23.326679 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8\": container with ID starting with cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8 not found: ID does not exist" containerID="cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8" Sep 16 13:45:23.326764 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.326711 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8"} err="failed to get container status \"cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8\": rpc error: code = NotFound desc = could not find container \"cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8\": container with ID starting with cd4aa9c3819098a89329dd57b4793ea30a28a05a5e1d9d36040a50fc4c8fd4e8 not found: ID does not exist" Sep 16 13:45:23.341139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.341110 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz"] Sep 16 13:45:23.355572 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.355539 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b0514e-7bd76cc595-m25rz"] Sep 16 13:45:23.388069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.388038 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa1a710b-dc96-455e-bfd1-80c3b55128b0" path="/var/lib/kubelet/pods/aa1a710b-dc96-455e-bfd1-80c3b55128b0/volumes" Sep 16 13:45:23.388576 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.388558 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" path="/var/lib/kubelet/pods/c9982d77-4da2-4a7b-8af8-a2bbfb8eab23/volumes" Sep 16 13:45:23.467743 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.467709 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:23.565134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:23.565052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:24.589276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:24.589234 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:25.380245 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:25.380201 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:26.361509 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:26.361475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:29.972256 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:29.972215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:30.076130 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.076099 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:30.255796 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.255723 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd"] Sep 16 13:45:30.255957 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.255934 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" containerID="cri-o://658871916f2f1e3b0e967f0297ca29eccf00d257336e411417659c828cfb8ab0" gracePeriod=30 Sep 16 13:45:30.446667 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.446636 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:30.662123 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662041 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl"] Sep 16 13:45:30.662481 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662468 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa1a710b-dc96-455e-bfd1-80c3b55128b0" containerName="revoke-keys" Sep 16 13:45:30.662538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662483 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa1a710b-dc96-455e-bfd1-80c3b55128b0" containerName="revoke-keys" Sep 16 13:45:30.662538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662499 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" containerName="payload-processing" Sep 16 13:45:30.662538 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662504 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" containerName="payload-processing" Sep 16 13:45:30.662630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662566 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9982d77-4da2-4a7b-8af8-a2bbfb8eab23" containerName="payload-processing" Sep 16 13:45:30.662630 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.662573 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa1a710b-dc96-455e-bfd1-80c3b55128b0" containerName="revoke-keys" Sep 16 13:45:30.667076 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.667059 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.669615 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.669585 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:45:30.669745 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.669714 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:45:30.669745 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.669729 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:45:30.676823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.676795 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl"] Sep 16 13:45:30.770263 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.770217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f97367c-40c4-4a3a-a329-7b57d2f50f97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.770439 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.770295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnvb\" (UniqueName: \"kubernetes.io/projected/9f97367c-40c4-4a3a-a329-7b57d2f50f97-kube-api-access-4wnvb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.871094 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.871061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnvb\" (UniqueName: \"kubernetes.io/projected/9f97367c-40c4-4a3a-a329-7b57d2f50f97-kube-api-access-4wnvb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.871319 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.871173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f97367c-40c4-4a3a-a329-7b57d2f50f97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.871803 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.871784 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f97367c-40c4-4a3a-a329-7b57d2f50f97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.880268 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.880241 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnvb\" (UniqueName: \"kubernetes.io/projected/9f97367c-40c4-4a3a-a329-7b57d2f50f97-kube-api-access-4wnvb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:30.978608 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:30.978579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:31.259070 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:31.259038 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl"] Sep 16 13:45:31.260018 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:45:31.259996 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f97367c_40c4_4a3a_a329_7b57d2f50f97.slice/crio-73b281c64e5c2cca26ae349db284a4e8ebdad6ac8ded8f4c43590a01ea6b31c6 WatchSource:0}: Error finding container 73b281c64e5c2cca26ae349db284a4e8ebdad6ac8ded8f4c43590a01ea6b31c6: Status 404 returned error can't find the container with id 73b281c64e5c2cca26ae349db284a4e8ebdad6ac8ded8f4c43590a01ea6b31c6 Sep 16 13:45:31.376237 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:31.376199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" event={"ID":"9f97367c-40c4-4a3a-a329-7b57d2f50f97","Type":"ContainerStarted","Data":"73b281c64e5c2cca26ae349db284a4e8ebdad6ac8ded8f4c43590a01ea6b31c6"} Sep 16 13:45:31.465543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:31.465504 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:32.384527 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:32.384440 2581 generic.go:356] "Generic (PLEG): container finished" podID="9f97367c-40c4-4a3a-a329-7b57d2f50f97" containerID="f0ab33610391ce7ad89c2f3ae43c13353c200b46645088cf191318d0d03813cd" exitCode=0 Sep 16 13:45:32.384893 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:32.384530 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" event={"ID":"9f97367c-40c4-4a3a-a329-7b57d2f50f97","Type":"ContainerDied","Data":"f0ab33610391ce7ad89c2f3ae43c13353c200b46645088cf191318d0d03813cd"} Sep 16 13:45:32.508397 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:32.508367 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:33.330189 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.330146 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:33.533010 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.532985 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:33.592558 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.592477 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f97367c-40c4-4a3a-a329-7b57d2f50f97-maas-api-service-ca\") pod \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " Sep 16 13:45:33.592558 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.592510 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wnvb\" (UniqueName: \"kubernetes.io/projected/9f97367c-40c4-4a3a-a329-7b57d2f50f97-kube-api-access-4wnvb\") pod \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\" (UID: \"9f97367c-40c4-4a3a-a329-7b57d2f50f97\") " Sep 16 13:45:33.592859 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.592837 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f97367c-40c4-4a3a-a329-7b57d2f50f97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f97367c-40c4-4a3a-a329-7b57d2f50f97" (UID: "9f97367c-40c4-4a3a-a329-7b57d2f50f97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:33.594506 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.594486 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f97367c-40c4-4a3a-a329-7b57d2f50f97-kube-api-access-4wnvb" (OuterVolumeSpecName: "kube-api-access-4wnvb") pod "9f97367c-40c4-4a3a-a329-7b57d2f50f97" (UID: "9f97367c-40c4-4a3a-a329-7b57d2f50f97"). InnerVolumeSpecName "kube-api-access-4wnvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:33.693712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.693681 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f97367c-40c4-4a3a-a329-7b57d2f50f97-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:33.693712 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:33.693707 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnvb\" (UniqueName: \"kubernetes.io/projected/9f97367c-40c4-4a3a-a329-7b57d2f50f97-kube-api-access-4wnvb\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:34.078940 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:34.078902 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:34.403830 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:34.403745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" event={"ID":"9f97367c-40c4-4a3a-a329-7b57d2f50f97","Type":"ContainerDied","Data":"73b281c64e5c2cca26ae349db284a4e8ebdad6ac8ded8f4c43590a01ea6b31c6"} Sep 16 13:45:34.403830 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:34.403778 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl" Sep 16 13:45:34.403830 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:34.403791 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b281c64e5c2cca26ae349db284a4e8ebdad6ac8ded8f4c43590a01ea6b31c6" Sep 16 13:45:35.416124 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.416088 2581 generic.go:356] "Generic (PLEG): container finished" podID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerID="658871916f2f1e3b0e967f0297ca29eccf00d257336e411417659c828cfb8ab0" exitCode=0 Sep 16 13:45:35.416560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.416158 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" event={"ID":"ada1219c-edb8-4cce-81b0-f30a866f9ed1","Type":"ContainerDied","Data":"658871916f2f1e3b0e967f0297ca29eccf00d257336e411417659c828cfb8ab0"} Sep 16 13:45:35.416560 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.416222 2581 scope.go:117] "RemoveContainer" containerID="35fddaeda145991bc8b42607e1b0bb6bfba6a3f71e9f9cd2dcfac817ef697bc3" Sep 16 13:45:35.515657 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.515591 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:45:35.611702 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611671 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-envoy\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.611866 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611741 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istiod-ca-cert\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.611866 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611783 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-data\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.611866 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611816 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btrg2\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-kube-api-access-btrg2\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.611866 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611843 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-certs\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.612102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611869 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-socket\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.612102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611904 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-credential-socket\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.612102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611959 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-podinfo\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.612102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.611985 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-token\") pod \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\" (UID: \"ada1219c-edb8-4cce-81b0-f30a866f9ed1\") " Sep 16 13:45:35.612102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612055 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-data" (OuterVolumeSpecName: "istio-data") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:35.612389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612204 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:35.612389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612314 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:35.612389 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612322 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:35.612525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612392 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-data\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.612525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612406 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-certs\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.612773 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.612742 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:35.613909 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.613886 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:45:35.614106 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.614090 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-token" (OuterVolumeSpecName: "istio-token") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:35.614601 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.614580 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:45:35.614695 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.614608 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-kube-api-access-btrg2" (OuterVolumeSpecName: "kube-api-access-btrg2") pod "ada1219c-edb8-4cce-81b0-f30a866f9ed1" (UID: "ada1219c-edb8-4cce-81b0-f30a866f9ed1"). InnerVolumeSpecName "kube-api-access-btrg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:35.713842 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713817 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-credential-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.713842 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713841 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-podinfo\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.713986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713851 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-token\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.713986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713859 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istio-envoy\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.713986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713867 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada1219c-edb8-4cce-81b0-f30a866f9ed1-istiod-ca-cert\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.713986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713875 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrg2\" (UniqueName: \"kubernetes.io/projected/ada1219c-edb8-4cce-81b0-f30a866f9ed1-kube-api-access-btrg2\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.713986 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.713884 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada1219c-edb8-4cce-81b0-f30a866f9ed1-workload-socket\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:35.850783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:35.850705 2581 scope.go:117] "RemoveContainer" containerID="658871916f2f1e3b0e967f0297ca29eccf00d257336e411417659c828cfb8ab0" Sep 16 13:45:36.422392 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:36.422362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" Sep 16 13:45:36.422392 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:36.422386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd" event={"ID":"ada1219c-edb8-4cce-81b0-f30a866f9ed1","Type":"ContainerDied","Data":"27b4664f06be1c9bcc875f6951f401611c3edbcb27a4381f86298f5fb9529347"} Sep 16 13:45:36.489186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:36.489161 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd"] Sep 16 13:45:36.494646 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:36.494618 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0514e-openshift-default-79b785f87b-lrpzd"] Sep 16 13:45:37.388119 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:37.388077 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" path="/var/lib/kubelet/pods/ada1219c-edb8-4cce-81b0-f30a866f9ed1/volumes" Sep 16 13:45:41.261809 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.261772 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:41.264749 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.264722 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt"] Sep 16 13:45:41.267625 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.267602 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7841d9-29826105-lbtwt"] Sep 16 13:45:41.298845 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.298813 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74"] Sep 16 13:45:41.299055 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.299023 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" podUID="fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" containerName="payload-processing" containerID="cri-o://f83f315ecfdd55e652aa4794f2a7e4d6c5dc656ee70d056bd2c9cf1ef562c919" gracePeriod=30 Sep 16 13:45:41.350437 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.350404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:41.387893 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.387852 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2437b5eb-2e02-4764-991b-d5561d58f4b2" path="/var/lib/kubelet/pods/2437b5eb-2e02-4764-991b-d5561d58f4b2/volumes" Sep 16 13:45:41.456111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.456080 2581 generic.go:356] "Generic (PLEG): container finished" podID="fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" containerID="f83f315ecfdd55e652aa4794f2a7e4d6c5dc656ee70d056bd2c9cf1ef562c919" exitCode=0 Sep 16 13:45:41.456327 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.456186 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" event={"ID":"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c","Type":"ContainerDied","Data":"f83f315ecfdd55e652aa4794f2a7e4d6c5dc656ee70d056bd2c9cf1ef562c919"} Sep 16 13:45:41.456422 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.456355 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:41.502596 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.502554 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl"] Sep 16 13:45:41.510954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.510924 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7841d9-pkccl"] Sep 16 13:45:41.556556 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.556518 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:41.567274 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.567254 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:45:41.664823 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.664790 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-plugins-config-volume\") pod \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " Sep 16 13:45:41.665002 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.664879 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b6fh\" (UniqueName: \"kubernetes.io/projected/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-kube-api-access-2b6fh\") pod \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\" (UID: \"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c\") " Sep 16 13:45:41.665174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.665152 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" (UID: "fe13d5bd-0ea7-4e26-a122-e173d0a2f78c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:41.666906 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.666879 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-kube-api-access-2b6fh" (OuterVolumeSpecName: "kube-api-access-2b6fh") pod "fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" (UID: "fe13d5bd-0ea7-4e26-a122-e173d0a2f78c"). InnerVolumeSpecName "kube-api-access-2b6fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:41.766518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.766489 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:41.766518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.766512 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b6fh\" (UniqueName: \"kubernetes.io/projected/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c-kube-api-access-2b6fh\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:45:41.908946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:41.908855 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:42.007785 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.007747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:42.464183 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.464158 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" Sep 16 13:45:42.464656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.464156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74" event={"ID":"fe13d5bd-0ea7-4e26-a122-e173d0a2f78c","Type":"ContainerDied","Data":"9e3d068b28302e8780172d9764ccf24a03fd71563fefe83a7ffc115446435c10"} Sep 16 13:45:42.464656 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.464434 2581 scope.go:117] "RemoveContainer" containerID="f83f315ecfdd55e652aa4794f2a7e4d6c5dc656ee70d056bd2c9cf1ef562c919" Sep 16 13:45:42.488360 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.488331 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74"] Sep 16 13:45:42.493291 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.493262 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7841d9-8f947fcc-f5w74"] Sep 16 13:45:42.877212 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:42.877129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:43.390577 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:43.390542 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f97367c-40c4-4a3a-a329-7b57d2f50f97" path="/var/lib/kubelet/pods/9f97367c-40c4-4a3a-a329-7b57d2f50f97/volumes" Sep 16 13:45:43.390903 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:43.390889 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" path="/var/lib/kubelet/pods/fe13d5bd-0ea7-4e26-a122-e173d0a2f78c/volumes" Sep 16 13:45:43.629223 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:43.629194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:46.699820 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:46.699783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:46.841073 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:46.841037 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:48.848938 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:48.848900 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:48.930239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:48.930194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:45:49.146468 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:45:49.146380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:12.378740 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:46:12.378651 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:20.581460 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:46:20.581426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:21.030546 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:46:21.030511 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:21.841988 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:46:21.841949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:46:22.745319 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:46:22.745263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:21.868478 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:47:21.868440 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:47:55.512959 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:47:55.512863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:48:35.595953 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:48:35.595921 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:48:35.602793 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:48:35.602771 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:48:37.374351 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:48:37.374310 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:13.997152 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:49:13.997063 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:49:14.050014 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:49:14.049976 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:06.258828 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:50:06.258792 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:07.884598 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:50:07.884564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:11.979352 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:50:11.979308 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:50:35.937160 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:50:35.937088 2581 scope.go:117] "RemoveContainer" containerID="72ffdbf717803236a44a2b25d1c99a544daaa664db6a8554377b4a1d77a1dba9" Sep 16 13:50:38.633600 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:50:38.633561 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:51:35.960251 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:51:35.960217 2581 scope.go:117] "RemoveContainer" containerID="7d1cae88a350864e39d183f296ae4ef2359e01bbed59cbd3e3571fd35ef1c228" Sep 16 13:51:35.969446 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:51:35.969422 2581 scope.go:117] "RemoveContainer" containerID="f0ab33610391ce7ad89c2f3ae43c13353c200b46645088cf191318d0d03813cd" Sep 16 13:51:35.977583 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:51:35.977567 2581 scope.go:117] "RemoveContainer" containerID="3f02d24defc5c92da46c43409e0d091330d2e68a0e5e2e8e872428031d6c3869" Sep 16 13:51:35.985616 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:51:35.985595 2581 scope.go:117] "RemoveContainer" containerID="979a94027d499a671c48f23fb68aca4f872d83c184890313c42358812eece692" Sep 16 13:52:22.599186 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:52:22.599149 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:52:56.230882 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:52:56.230846 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:10.230819 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:53:10.230787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:53:35.635562 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:53:35.635492 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:53:35.650229 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:53:35.650207 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:53:38.318224 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:53:38.318188 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:17.358684 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:17.358640 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:18.334207 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.334163 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:18.438497 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438460 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7"] Sep 16 13:54:18.438896 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438882 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" Sep 16 13:54:18.438946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438899 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" Sep 16 13:54:18.438946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438909 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f97367c-40c4-4a3a-a329-7b57d2f50f97" containerName="revoke-keys" Sep 16 13:54:18.438946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438915 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f97367c-40c4-4a3a-a329-7b57d2f50f97" containerName="revoke-keys" Sep 16 13:54:18.438946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438936 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" containerName="payload-processing" Sep 16 13:54:18.439084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.438945 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" containerName="payload-processing" Sep 16 13:54:18.439084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.439034 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe13d5bd-0ea7-4e26-a122-e173d0a2f78c" containerName="payload-processing" Sep 16 13:54:18.439084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.439043 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" Sep 16 13:54:18.439084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.439050 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f97367c-40c4-4a3a-a329-7b57d2f50f97" containerName="revoke-keys" Sep 16 13:54:18.439084 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.439058 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" Sep 16 13:54:18.442195 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.442179 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.444647 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.444621 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:54:18.444792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.444657 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:54:18.444792 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.444634 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:54:18.449766 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.449747 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7"] Sep 16 13:54:18.489863 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.489832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bd0034-bf01-45da-b2c7-984e7810d3f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.490022 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.489877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kw74\" (UniqueName: \"kubernetes.io/projected/95bd0034-bf01-45da-b2c7-984e7810d3f6-kube-api-access-8kw74\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.591427 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.591351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bd0034-bf01-45da-b2c7-984e7810d3f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.591427 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.591394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kw74\" (UniqueName: \"kubernetes.io/projected/95bd0034-bf01-45da-b2c7-984e7810d3f6-kube-api-access-8kw74\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.591954 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.591935 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bd0034-bf01-45da-b2c7-984e7810d3f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.599579 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.599556 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kw74\" (UniqueName: \"kubernetes.io/projected/95bd0034-bf01-45da-b2c7-984e7810d3f6-kube-api-access-8kw74\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:18.744788 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.744752 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:18.755154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:18.755132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:19.032457 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:19.032429 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7"] Sep 16 13:54:19.033631 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:54:19.033603 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bd0034_bf01_45da_b2c7_984e7810d3f6.slice/crio-6a3efdbfccd95db66b1672907bee6a1e14ff348a56f03d8c15539c034ece6e13 WatchSource:0}: Error finding container 6a3efdbfccd95db66b1672907bee6a1e14ff348a56f03d8c15539c034ece6e13: Status 404 returned error can't find the container with id 6a3efdbfccd95db66b1672907bee6a1e14ff348a56f03d8c15539c034ece6e13 Sep 16 13:54:19.624739 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:19.624652 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:19.633841 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:19.633807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" event={"ID":"95bd0034-bf01-45da-b2c7-984e7810d3f6","Type":"ContainerStarted","Data":"315d6d1898fc1d62ca6a864444b3133f856656b19465a02c92fc8bcd66008e54"} Sep 16 13:54:19.633970 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:19.633866 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" event={"ID":"95bd0034-bf01-45da-b2c7-984e7810d3f6","Type":"ContainerStarted","Data":"6a3efdbfccd95db66b1672907bee6a1e14ff348a56f03d8c15539c034ece6e13"} Sep 16 13:54:19.651439 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:19.651399 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" podStartSLOduration=1.651384365 podStartE2EDuration="1.651384365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:54:19.64926054 +0000 UTC m=+2744.768987131" watchObservedRunningTime="2026-09-16 13:54:19.651384365 +0000 UTC m=+2744.771110915" Sep 16 13:54:20.641733 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:20.641701 2581 generic.go:356] "Generic (PLEG): container finished" podID="95bd0034-bf01-45da-b2c7-984e7810d3f6" containerID="315d6d1898fc1d62ca6a864444b3133f856656b19465a02c92fc8bcd66008e54" exitCode=0 Sep 16 13:54:20.642114 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:20.641780 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" event={"ID":"95bd0034-bf01-45da-b2c7-984e7810d3f6","Type":"ContainerDied","Data":"315d6d1898fc1d62ca6a864444b3133f856656b19465a02c92fc8bcd66008e54"} Sep 16 13:54:20.644555 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:20.644525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:21.536683 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:21.536648 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:21.786043 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:21.786015 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:21.919582 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:21.919496 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kw74\" (UniqueName: \"kubernetes.io/projected/95bd0034-bf01-45da-b2c7-984e7810d3f6-kube-api-access-8kw74\") pod \"95bd0034-bf01-45da-b2c7-984e7810d3f6\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " Sep 16 13:54:21.919783 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:21.919589 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bd0034-bf01-45da-b2c7-984e7810d3f6-maas-api-service-ca\") pod \"95bd0034-bf01-45da-b2c7-984e7810d3f6\" (UID: \"95bd0034-bf01-45da-b2c7-984e7810d3f6\") " Sep 16 13:54:21.919946 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:21.919918 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95bd0034-bf01-45da-b2c7-984e7810d3f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95bd0034-bf01-45da-b2c7-984e7810d3f6" (UID: "95bd0034-bf01-45da-b2c7-984e7810d3f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:54:21.921766 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:21.921744 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bd0034-bf01-45da-b2c7-984e7810d3f6-kube-api-access-8kw74" (OuterVolumeSpecName: "kube-api-access-8kw74") pod "95bd0034-bf01-45da-b2c7-984e7810d3f6" (UID: "95bd0034-bf01-45da-b2c7-984e7810d3f6"). InnerVolumeSpecName "kube-api-access-8kw74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:54:22.021030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:22.020998 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kw74\" (UniqueName: \"kubernetes.io/projected/95bd0034-bf01-45da-b2c7-984e7810d3f6-kube-api-access-8kw74\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:22.021030 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:22.021026 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95bd0034-bf01-45da-b2c7-984e7810d3f6-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:22.655228 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:22.655201 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" Sep 16 13:54:22.655431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:22.655242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7" event={"ID":"95bd0034-bf01-45da-b2c7-984e7810d3f6","Type":"ContainerDied","Data":"6a3efdbfccd95db66b1672907bee6a1e14ff348a56f03d8c15539c034ece6e13"} Sep 16 13:54:22.655431 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:22.655311 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a3efdbfccd95db66b1672907bee6a1e14ff348a56f03d8c15539c034ece6e13" Sep 16 13:54:24.608034 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:24.608000 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:24.712795 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:24.712758 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:28.940717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:28.940677 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:28.942336 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:28.942309 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8"] Sep 16 13:54:28.946513 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:28.946488 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7e7bb4-29826105-glqw8"] Sep 16 13:54:29.038134 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.038096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:29.138171 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.138137 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:29.147110 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.147083 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7"] Sep 16 13:54:29.149784 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.149766 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7e7bb4-jj6h7"] Sep 16 13:54:29.238199 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.238156 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:29.387216 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.387184 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed7548e-6771-46e8-b3ba-e35a4e81f8d8" path="/var/lib/kubelet/pods/1ed7548e-6771-46e8-b3ba-e35a4e81f8d8/volumes" Sep 16 13:54:29.387603 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.387588 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95bd0034-bf01-45da-b2c7-984e7810d3f6" path="/var/lib/kubelet/pods/95bd0034-bf01-45da-b2c7-984e7810d3f6/volumes" Sep 16 13:54:29.541139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.541054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:29.641732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:29.641701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:30.448464 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:30.448433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:35.506772 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:35.506737 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:35.608523 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:35.608486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:35.796277 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:35.796190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:36.036355 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.036322 2581 scope.go:117] "RemoveContainer" containerID="3ac148e2b4c4dda54e6774ee13acd0ac4c9709145cdebeaba32500fcc9a6b308" Sep 16 13:54:36.128396 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.128371 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv"] Sep 16 13:54:36.128877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.128858 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bd0034-bf01-45da-b2c7-984e7810d3f6" containerName="revoke-keys" Sep 16 13:54:36.128950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.128880 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bd0034-bf01-45da-b2c7-984e7810d3f6" containerName="revoke-keys" Sep 16 13:54:36.128950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.128894 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" Sep 16 13:54:36.128950 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.128899 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada1219c-edb8-4cce-81b0-f30a866f9ed1" containerName="istio-proxy" Sep 16 13:54:36.129051 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.128964 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95bd0034-bf01-45da-b2c7-984e7810d3f6" containerName="revoke-keys" Sep 16 13:54:36.133097 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.133081 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.135463 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.135447 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:54:36.136154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.136138 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:54:36.136230 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.136138 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 13:54:36.141911 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.141886 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv"] Sep 16 13:54:36.248619 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.248580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43dd47a-7019-4896-ad2b-428df37c36aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.248801 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.248649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67h9v\" (UniqueName: \"kubernetes.io/projected/b43dd47a-7019-4896-ad2b-428df37c36aa-kube-api-access-67h9v\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.302543 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.302450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:36.349636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.349469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43dd47a-7019-4896-ad2b-428df37c36aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.349636 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.349555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67h9v\" (UniqueName: \"kubernetes.io/projected/b43dd47a-7019-4896-ad2b-428df37c36aa-kube-api-access-67h9v\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.350276 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.350257 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43dd47a-7019-4896-ad2b-428df37c36aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.357703 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.357684 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67h9v\" (UniqueName: \"kubernetes.io/projected/b43dd47a-7019-4896-ad2b-428df37c36aa-kube-api-access-67h9v\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.446192 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.446159 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:36.723013 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.722977 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv"] Sep 16 13:54:36.723946 ip-10-0-133-73 kubenswrapper[2581]: W0916 13:54:36.723923 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43dd47a_7019_4896_ad2b_428df37c36aa.slice/crio-411cf729a8488ace282e24d56e43a1d9c50362d4886bf9e9bcc63d586963f43b WatchSource:0}: Error finding container 411cf729a8488ace282e24d56e43a1d9c50362d4886bf9e9bcc63d586963f43b: Status 404 returned error can't find the container with id 411cf729a8488ace282e24d56e43a1d9c50362d4886bf9e9bcc63d586963f43b Sep 16 13:54:36.740122 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:36.740094 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" event={"ID":"b43dd47a-7019-4896-ad2b-428df37c36aa","Type":"ContainerStarted","Data":"411cf729a8488ace282e24d56e43a1d9c50362d4886bf9e9bcc63d586963f43b"} Sep 16 13:54:37.131115 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:37.131079 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:37.747139 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:37.747105 2581 generic.go:356] "Generic (PLEG): container finished" podID="b43dd47a-7019-4896-ad2b-428df37c36aa" containerID="6027086f4798b5979de1b024218ac93418828073d653771b36e32c706e7ed612" exitCode=0 Sep 16 13:54:37.747531 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:37.747144 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" event={"ID":"b43dd47a-7019-4896-ad2b-428df37c36aa","Type":"ContainerDied","Data":"6027086f4798b5979de1b024218ac93418828073d653771b36e32c706e7ed612"} Sep 16 13:54:37.953382 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:37.953349 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:38.892111 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:38.892087 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:38.971720 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:38.971693 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67h9v\" (UniqueName: \"kubernetes.io/projected/b43dd47a-7019-4896-ad2b-428df37c36aa-kube-api-access-67h9v\") pod \"b43dd47a-7019-4896-ad2b-428df37c36aa\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " Sep 16 13:54:38.971869 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:38.971808 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43dd47a-7019-4896-ad2b-428df37c36aa-maas-api-service-ca\") pod \"b43dd47a-7019-4896-ad2b-428df37c36aa\" (UID: \"b43dd47a-7019-4896-ad2b-428df37c36aa\") " Sep 16 13:54:38.972110 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:38.972089 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43dd47a-7019-4896-ad2b-428df37c36aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b43dd47a-7019-4896-ad2b-428df37c36aa" (UID: "b43dd47a-7019-4896-ad2b-428df37c36aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:54:38.973734 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:38.973714 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43dd47a-7019-4896-ad2b-428df37c36aa-kube-api-access-67h9v" (OuterVolumeSpecName: "kube-api-access-67h9v") pod "b43dd47a-7019-4896-ad2b-428df37c36aa" (UID: "b43dd47a-7019-4896-ad2b-428df37c36aa"). InnerVolumeSpecName "kube-api-access-67h9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:54:39.073154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:39.073090 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b43dd47a-7019-4896-ad2b-428df37c36aa-maas-api-service-ca\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:39.073154 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:39.073114 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67h9v\" (UniqueName: \"kubernetes.io/projected/b43dd47a-7019-4896-ad2b-428df37c36aa-kube-api-access-67h9v\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:39.761525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:39.761497 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" Sep 16 13:54:39.761525 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:39.761507 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv" event={"ID":"b43dd47a-7019-4896-ad2b-428df37c36aa","Type":"ContainerDied","Data":"411cf729a8488ace282e24d56e43a1d9c50362d4886bf9e9bcc63d586963f43b"} Sep 16 13:54:39.761810 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:39.761552 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="411cf729a8488ace282e24d56e43a1d9c50362d4886bf9e9bcc63d586963f43b" Sep 16 13:54:46.624238 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.624188 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms"] Sep 16 13:54:46.627172 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.627147 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8c3cdf-29826105-4wqms"] Sep 16 13:54:46.629862 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.629829 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:46.651795 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.651768 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj"] Sep 16 13:54:46.652069 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.652038 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" podUID="0d893da2-e298-4a92-9f13-328202499b97" containerName="payload-processing" containerID="cri-o://ba502b06d947fe238fbcb08e9670d0b9733483a18667269a57147d4352442ab8" gracePeriod=30 Sep 16 13:54:46.662698 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.662678 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn"] Sep 16 13:54:46.662944 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.662915 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" podUID="fb749133-4021-438e-b6f3-611bf776f67a" containerName="payload-pre-processing" containerID="cri-o://3575c1f0f138abb1bb9812ba38a0bedf56f948d221de4b3a070810a34ae9d592" gracePeriod=30 Sep 16 13:54:46.697425 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.697387 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:46.805877 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.805846 2581 generic.go:356] "Generic (PLEG): container finished" podID="0d893da2-e298-4a92-9f13-328202499b97" containerID="ba502b06d947fe238fbcb08e9670d0b9733483a18667269a57147d4352442ab8" exitCode=0 Sep 16 13:54:46.806071 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.805954 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" event={"ID":"0d893da2-e298-4a92-9f13-328202499b97","Type":"ContainerDied","Data":"ba502b06d947fe238fbcb08e9670d0b9733483a18667269a57147d4352442ab8"} Sep 16 13:54:46.806694 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.806655 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:46.808275 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.808236 2581 generic.go:356] "Generic (PLEG): container finished" podID="fb749133-4021-438e-b6f3-611bf776f67a" containerID="3575c1f0f138abb1bb9812ba38a0bedf56f948d221de4b3a070810a34ae9d592" exitCode=0 Sep 16 13:54:46.808410 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.808303 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" event={"ID":"fb749133-4021-438e-b6f3-611bf776f67a","Type":"ContainerDied","Data":"3575c1f0f138abb1bb9812ba38a0bedf56f948d221de4b3a070810a34ae9d592"} Sep 16 13:54:46.890174 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.890079 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv"] Sep 16 13:54:46.900065 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.900038 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8c3cdf-vd4nv"] Sep 16 13:54:46.945597 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.945576 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:54:46.948398 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:46.948381 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:54:47.018235 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.018181 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:47.045875 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.045845 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkbb8\" (UniqueName: \"kubernetes.io/projected/0d893da2-e298-4a92-9f13-328202499b97-kube-api-access-zkbb8\") pod \"0d893da2-e298-4a92-9f13-328202499b97\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " Sep 16 13:54:47.046044 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.045901 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g576s\" (UniqueName: \"kubernetes.io/projected/fb749133-4021-438e-b6f3-611bf776f67a-kube-api-access-g576s\") pod \"fb749133-4021-438e-b6f3-611bf776f67a\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " Sep 16 13:54:47.048012 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.047984 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d893da2-e298-4a92-9f13-328202499b97-kube-api-access-zkbb8" (OuterVolumeSpecName: "kube-api-access-zkbb8") pod "0d893da2-e298-4a92-9f13-328202499b97" (UID: "0d893da2-e298-4a92-9f13-328202499b97"). InnerVolumeSpecName "kube-api-access-zkbb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:54:47.048102 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.048055 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb749133-4021-438e-b6f3-611bf776f67a-kube-api-access-g576s" (OuterVolumeSpecName: "kube-api-access-g576s") pod "fb749133-4021-438e-b6f3-611bf776f67a" (UID: "fb749133-4021-438e-b6f3-611bf776f67a"). InnerVolumeSpecName "kube-api-access-g576s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:54:47.146521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.146443 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d893da2-e298-4a92-9f13-328202499b97-plugins-config-volume\") pod \"0d893da2-e298-4a92-9f13-328202499b97\" (UID: \"0d893da2-e298-4a92-9f13-328202499b97\") " Sep 16 13:54:47.146521 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.146487 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb749133-4021-438e-b6f3-611bf776f67a-plugins-config-volume\") pod \"fb749133-4021-438e-b6f3-611bf776f67a\" (UID: \"fb749133-4021-438e-b6f3-611bf776f67a\") " Sep 16 13:54:47.146732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.146684 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkbb8\" (UniqueName: \"kubernetes.io/projected/0d893da2-e298-4a92-9f13-328202499b97-kube-api-access-zkbb8\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:47.146732 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.146695 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g576s\" (UniqueName: \"kubernetes.io/projected/fb749133-4021-438e-b6f3-611bf776f67a-kube-api-access-g576s\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:47.146843 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.146817 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d893da2-e298-4a92-9f13-328202499b97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d893da2-e298-4a92-9f13-328202499b97" (UID: "0d893da2-e298-4a92-9f13-328202499b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:54:47.146934 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.146913 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb749133-4021-438e-b6f3-611bf776f67a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb749133-4021-438e-b6f3-611bf776f67a" (UID: "fb749133-4021-438e-b6f3-611bf776f67a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:54:47.247956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.247922 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb749133-4021-438e-b6f3-611bf776f67a-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:47.247956 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.247951 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d893da2-e298-4a92-9f13-328202499b97-plugins-config-volume\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 13:54:47.309766 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.309726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:47.387831 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.387794 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a377ef0-11c7-4c79-b929-e81de1893e36" path="/var/lib/kubelet/pods/8a377ef0-11c7-4c79-b929-e81de1893e36/volumes" Sep 16 13:54:47.388428 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.388409 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b43dd47a-7019-4896-ad2b-428df37c36aa" path="/var/lib/kubelet/pods/b43dd47a-7019-4896-ad2b-428df37c36aa/volumes" Sep 16 13:54:47.425078 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.425054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:47.815655 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.815620 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" event={"ID":"0d893da2-e298-4a92-9f13-328202499b97","Type":"ContainerDied","Data":"cc4adc2b46656a7b15c0487a72d30102881da8545be0ec58141a0f49ba6a5e50"} Sep 16 13:54:47.816088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.815635 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj" Sep 16 13:54:47.816088 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.815683 2581 scope.go:117] "RemoveContainer" containerID="ba502b06d947fe238fbcb08e9670d0b9733483a18667269a57147d4352442ab8" Sep 16 13:54:47.817255 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.817233 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" Sep 16 13:54:47.817364 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.817254 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn" event={"ID":"fb749133-4021-438e-b6f3-611bf776f67a","Type":"ContainerDied","Data":"9cfc9f3ee0203e5ddbc1946c266df74cedcf4379b87b38d16662fc6ed9a9db30"} Sep 16 13:54:47.825362 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.825334 2581 scope.go:117] "RemoveContainer" containerID="3575c1f0f138abb1bb9812ba38a0bedf56f948d221de4b3a070810a34ae9d592" Sep 16 13:54:47.835717 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.835698 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj"] Sep 16 13:54:47.840452 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.840432 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8c3cdf-8f9d7694c-g4mpj"] Sep 16 13:54:47.850497 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.850475 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn"] Sep 16 13:54:47.854239 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:47.854220 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8c3cdf-5c9f8cd8db-zksqn"] Sep 16 13:54:49.387894 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:49.387858 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d893da2-e298-4a92-9f13-328202499b97" path="/var/lib/kubelet/pods/0d893da2-e298-4a92-9f13-328202499b97/volumes" Sep 16 13:54:49.388363 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:49.388217 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb749133-4021-438e-b6f3-611bf776f67a" path="/var/lib/kubelet/pods/fb749133-4021-438e-b6f3-611bf776f67a/volumes" Sep 16 13:54:53.147507 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:53.147473 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:53.208943 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:53.208902 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:54:53.420141 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:54:53.420065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:55:36.059357 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:55:36.059327 2581 scope.go:117] "RemoveContainer" containerID="a1d9798364009ddbef9e2dc2fb13f3f115ef06cfd8ba07ceafbb965dbfdb8132" Sep 16 13:56:29.010115 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:56:29.010010 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:56:30.868518 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:56:30.868479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:14.232568 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:58:14.232482 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:58:35.676861 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:58:35.676834 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:58:35.691273 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:58:35.691252 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 13:59:28.757313 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:59:28.757223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:54.191105 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:59:54.191070 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:54.263077 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:59:54.263048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 13:59:58.433799 ip-10-0-133-73 kubenswrapper[2581]: I0916 13:59:58.433766 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 14:00:00.177440 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.177409 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t"] Sep 16 14:00:00.178026 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.177993 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43dd47a-7019-4896-ad2b-428df37c36aa" containerName="revoke-keys" Sep 16 14:00:00.178026 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178015 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43dd47a-7019-4896-ad2b-428df37c36aa" containerName="revoke-keys" Sep 16 14:00:00.178155 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178031 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d893da2-e298-4a92-9f13-328202499b97" containerName="payload-processing" Sep 16 14:00:00.178155 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178040 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d893da2-e298-4a92-9f13-328202499b97" containerName="payload-processing" Sep 16 14:00:00.178155 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178074 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb749133-4021-438e-b6f3-611bf776f67a" containerName="payload-pre-processing" Sep 16 14:00:00.178155 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178084 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb749133-4021-438e-b6f3-611bf776f67a" containerName="payload-pre-processing" Sep 16 14:00:00.178400 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178199 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d893da2-e298-4a92-9f13-328202499b97" containerName="payload-processing" Sep 16 14:00:00.178400 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178215 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43dd47a-7019-4896-ad2b-428df37c36aa" containerName="revoke-keys" Sep 16 14:00:00.178400 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.178225 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb749133-4021-438e-b6f3-611bf776f67a" containerName="payload-pre-processing" Sep 16 14:00:00.181460 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.181438 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh"] Sep 16 14:00:00.181589 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.181552 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:00.185263 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.185244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:00.186021 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.186004 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 14:00:00.186610 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.186593 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 14:00:00.186730 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.186714 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvp5m\"" Sep 16 14:00:00.193009 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.192988 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2"] Sep 16 14:00:00.196276 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.196258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:00.218312 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.218266 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t"] Sep 16 14:00:00.222121 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.222097 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh"] Sep 16 14:00:00.226305 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.226254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2"] Sep 16 14:00:00.327872 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.327840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4k8\" (UniqueName: \"kubernetes.io/projected/085b7f6b-5eb4-4354-978e-e13befdf67c8-kube-api-access-2n4k8\") pod \"maas-api-key-cleanup-29826120-qqs5t\" (UID: \"085b7f6b-5eb4-4354-978e-e13befdf67c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:00.327872 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.327881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ddp\" (UniqueName: \"kubernetes.io/projected/6f310810-88ab-4075-8eb7-f67f89d81696-kube-api-access-78ddp\") pod \"maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2\" (UID: \"6f310810-88ab-4075-8eb7-f67f89d81696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:00.328113 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.327904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs46j\" (UniqueName: \"kubernetes.io/projected/52d4cce9-3deb-4de3-8c00-2d73a374a9c5-kube-api-access-vs46j\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh\" (UID: \"52d4cce9-3deb-4de3-8c00-2d73a374a9c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:00.428595 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.428508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4k8\" (UniqueName: \"kubernetes.io/projected/085b7f6b-5eb4-4354-978e-e13befdf67c8-kube-api-access-2n4k8\") pod \"maas-api-key-cleanup-29826120-qqs5t\" (UID: \"085b7f6b-5eb4-4354-978e-e13befdf67c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:00.428595 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.428547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ddp\" (UniqueName: \"kubernetes.io/projected/6f310810-88ab-4075-8eb7-f67f89d81696-kube-api-access-78ddp\") pod \"maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2\" (UID: \"6f310810-88ab-4075-8eb7-f67f89d81696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:00.428595 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.428572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs46j\" (UniqueName: \"kubernetes.io/projected/52d4cce9-3deb-4de3-8c00-2d73a374a9c5-kube-api-access-vs46j\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh\" (UID: \"52d4cce9-3deb-4de3-8c00-2d73a374a9c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:00.438579 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.438547 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ddp\" (UniqueName: \"kubernetes.io/projected/6f310810-88ab-4075-8eb7-f67f89d81696-kube-api-access-78ddp\") pod \"maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2\" (UID: \"6f310810-88ab-4075-8eb7-f67f89d81696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:00.439109 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.439090 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs46j\" (UniqueName: \"kubernetes.io/projected/52d4cce9-3deb-4de3-8c00-2d73a374a9c5-kube-api-access-vs46j\") pod \"maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh\" (UID: \"52d4cce9-3deb-4de3-8c00-2d73a374a9c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:00.439230 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.439212 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4k8\" (UniqueName: \"kubernetes.io/projected/085b7f6b-5eb4-4354-978e-e13befdf67c8-kube-api-access-2n4k8\") pod \"maas-api-key-cleanup-29826120-qqs5t\" (UID: \"085b7f6b-5eb4-4354-978e-e13befdf67c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:00.494011 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.493975 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:00.499168 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.499149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:00.506952 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.506934 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:00.788364 ip-10-0-133-73 kubenswrapper[2581]: W0916 14:00:00.788331 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085b7f6b_5eb4_4354_978e_e13befdf67c8.slice/crio-d8c814ed24836a812d85c9ba6008dba2e386034a9d50f34d901e942696b80e89 WatchSource:0}: Error finding container d8c814ed24836a812d85c9ba6008dba2e386034a9d50f34d901e942696b80e89: Status 404 returned error can't find the container with id d8c814ed24836a812d85c9ba6008dba2e386034a9d50f34d901e942696b80e89 Sep 16 14:00:00.788963 ip-10-0-133-73 kubenswrapper[2581]: W0916 14:00:00.788940 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4cce9_3deb_4de3_8c00_2d73a374a9c5.slice/crio-fa6eb31e3ce4d073f7a6a45fac37f73f5e2c62ea57d829267aeb8adbf38e0629 WatchSource:0}: Error finding container fa6eb31e3ce4d073f7a6a45fac37f73f5e2c62ea57d829267aeb8adbf38e0629: Status 404 returned error can't find the container with id fa6eb31e3ce4d073f7a6a45fac37f73f5e2c62ea57d829267aeb8adbf38e0629 Sep 16 14:00:00.789565 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.789530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t"] Sep 16 14:00:00.796441 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.796418 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh"] Sep 16 14:00:00.806979 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:00.806961 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2"] Sep 16 14:00:00.811371 ip-10-0-133-73 kubenswrapper[2581]: W0916 14:00:00.811348 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f310810_88ab_4075_8eb7_f67f89d81696.slice/crio-1b4b0183373419992a6044c517278093a29c6bd9dbbc76b4c6214348a32e4e72 WatchSource:0}: Error finding container 1b4b0183373419992a6044c517278093a29c6bd9dbbc76b4c6214348a32e4e72: Status 404 returned error can't find the container with id 1b4b0183373419992a6044c517278093a29c6bd9dbbc76b4c6214348a32e4e72 Sep 16 14:00:01.666150 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.666114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" event={"ID":"6f310810-88ab-4075-8eb7-f67f89d81696","Type":"ContainerStarted","Data":"8b9627d71592be7293db9dec1b11ba19dd1e8296a65d0a4eb5aaa9e5093e4fcc"} Sep 16 14:00:01.666579 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.666174 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" event={"ID":"6f310810-88ab-4075-8eb7-f67f89d81696","Type":"ContainerStarted","Data":"1b4b0183373419992a6044c517278093a29c6bd9dbbc76b4c6214348a32e4e72"} Sep 16 14:00:01.667789 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.667766 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" event={"ID":"52d4cce9-3deb-4de3-8c00-2d73a374a9c5","Type":"ContainerStarted","Data":"7ebbb4073fe9c9a6396e6bd928dd5730b694a9f5324801c402cb9ed11db6bbef"} Sep 16 14:00:01.667909 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.667805 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" event={"ID":"52d4cce9-3deb-4de3-8c00-2d73a374a9c5","Type":"ContainerStarted","Data":"fa6eb31e3ce4d073f7a6a45fac37f73f5e2c62ea57d829267aeb8adbf38e0629"} Sep 16 14:00:01.669339 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.669313 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" event={"ID":"085b7f6b-5eb4-4354-978e-e13befdf67c8","Type":"ContainerStarted","Data":"fe79495aba3fc687a95cc3ab2c98a43f5955d6d7e1430c6f42fd2c219ab02352"} Sep 16 14:00:01.669439 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.669355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" event={"ID":"085b7f6b-5eb4-4354-978e-e13befdf67c8","Type":"ContainerStarted","Data":"d8c814ed24836a812d85c9ba6008dba2e386034a9d50f34d901e942696b80e89"} Sep 16 14:00:01.684835 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.684795 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" podStartSLOduration=1.684781219 podStartE2EDuration="1.684781219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.682750734 +0000 UTC m=+3086.802477307" watchObservedRunningTime="2026-09-16 14:00:01.684781219 +0000 UTC m=+3086.804507781" Sep 16 14:00:01.707658 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.707610 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" podStartSLOduration=1.707591555 podStartE2EDuration="1.707591555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.706098876 +0000 UTC m=+3086.825825449" watchObservedRunningTime="2026-09-16 14:00:01.707591555 +0000 UTC m=+3086.827318104" Sep 16 14:00:01.728134 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:01.728084 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" podStartSLOduration=1.728069509 podStartE2EDuration="1.728069509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.723577216 +0000 UTC m=+3086.843303764" watchObservedRunningTime="2026-09-16 14:00:01.728069509 +0000 UTC m=+3086.847796058" Sep 16 14:00:02.677642 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:02.677604 2581 generic.go:356] "Generic (PLEG): container finished" podID="6f310810-88ab-4075-8eb7-f67f89d81696" containerID="8b9627d71592be7293db9dec1b11ba19dd1e8296a65d0a4eb5aaa9e5093e4fcc" exitCode=0 Sep 16 14:00:02.678037 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:02.677648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" event={"ID":"6f310810-88ab-4075-8eb7-f67f89d81696","Type":"ContainerDied","Data":"8b9627d71592be7293db9dec1b11ba19dd1e8296a65d0a4eb5aaa9e5093e4fcc"} Sep 16 14:00:02.679309 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:02.679263 2581 generic.go:356] "Generic (PLEG): container finished" podID="52d4cce9-3deb-4de3-8c00-2d73a374a9c5" containerID="7ebbb4073fe9c9a6396e6bd928dd5730b694a9f5324801c402cb9ed11db6bbef" exitCode=0 Sep 16 14:00:02.679401 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:02.679344 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" event={"ID":"52d4cce9-3deb-4de3-8c00-2d73a374a9c5","Type":"ContainerDied","Data":"7ebbb4073fe9c9a6396e6bd928dd5730b694a9f5324801c402cb9ed11db6bbef"} Sep 16 14:00:02.680844 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:02.680822 2581 generic.go:356] "Generic (PLEG): container finished" podID="085b7f6b-5eb4-4354-978e-e13befdf67c8" containerID="fe79495aba3fc687a95cc3ab2c98a43f5955d6d7e1430c6f42fd2c219ab02352" exitCode=0 Sep 16 14:00:02.681020 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:02.680847 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" event={"ID":"085b7f6b-5eb4-4354-978e-e13befdf67c8","Type":"ContainerDied","Data":"fe79495aba3fc687a95cc3ab2c98a43f5955d6d7e1430c6f42fd2c219ab02352"} Sep 16 14:00:03.837975 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.837946 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:03.874892 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.874870 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:03.878059 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.878038 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:03.959666 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.959638 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78ddp\" (UniqueName: \"kubernetes.io/projected/6f310810-88ab-4075-8eb7-f67f89d81696-kube-api-access-78ddp\") pod \"6f310810-88ab-4075-8eb7-f67f89d81696\" (UID: \"6f310810-88ab-4075-8eb7-f67f89d81696\") " Sep 16 14:00:03.959831 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.959679 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n4k8\" (UniqueName: \"kubernetes.io/projected/085b7f6b-5eb4-4354-978e-e13befdf67c8-kube-api-access-2n4k8\") pod \"085b7f6b-5eb4-4354-978e-e13befdf67c8\" (UID: \"085b7f6b-5eb4-4354-978e-e13befdf67c8\") " Sep 16 14:00:03.959831 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.959735 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs46j\" (UniqueName: \"kubernetes.io/projected/52d4cce9-3deb-4de3-8c00-2d73a374a9c5-kube-api-access-vs46j\") pod \"52d4cce9-3deb-4de3-8c00-2d73a374a9c5\" (UID: \"52d4cce9-3deb-4de3-8c00-2d73a374a9c5\") " Sep 16 14:00:03.961784 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.961748 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d4cce9-3deb-4de3-8c00-2d73a374a9c5-kube-api-access-vs46j" (OuterVolumeSpecName: "kube-api-access-vs46j") pod "52d4cce9-3deb-4de3-8c00-2d73a374a9c5" (UID: "52d4cce9-3deb-4de3-8c00-2d73a374a9c5"). InnerVolumeSpecName "kube-api-access-vs46j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:00:03.961899 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.961796 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f310810-88ab-4075-8eb7-f67f89d81696-kube-api-access-78ddp" (OuterVolumeSpecName: "kube-api-access-78ddp") pod "6f310810-88ab-4075-8eb7-f67f89d81696" (UID: "6f310810-88ab-4075-8eb7-f67f89d81696"). InnerVolumeSpecName "kube-api-access-78ddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:00:03.961899 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:03.961837 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085b7f6b-5eb4-4354-978e-e13befdf67c8-kube-api-access-2n4k8" (OuterVolumeSpecName: "kube-api-access-2n4k8") pod "085b7f6b-5eb4-4354-978e-e13befdf67c8" (UID: "085b7f6b-5eb4-4354-978e-e13befdf67c8"). InnerVolumeSpecName "kube-api-access-2n4k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:00:04.061471 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.061439 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4k8\" (UniqueName: \"kubernetes.io/projected/085b7f6b-5eb4-4354-978e-e13befdf67c8-kube-api-access-2n4k8\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 14:00:04.061471 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.061465 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs46j\" (UniqueName: \"kubernetes.io/projected/52d4cce9-3deb-4de3-8c00-2d73a374a9c5-kube-api-access-vs46j\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 14:00:04.061471 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.061474 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ddp\" (UniqueName: \"kubernetes.io/projected/6f310810-88ab-4075-8eb7-f67f89d81696-kube-api-access-78ddp\") on node \"ip-10-0-133-73.ec2.internal\" DevicePath \"\"" Sep 16 14:00:04.695854 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.695820 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" Sep 16 14:00:04.696037 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.695822 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2" event={"ID":"6f310810-88ab-4075-8eb7-f67f89d81696","Type":"ContainerDied","Data":"1b4b0183373419992a6044c517278093a29c6bd9dbbc76b4c6214348a32e4e72"} Sep 16 14:00:04.696037 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.695943 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4b0183373419992a6044c517278093a29c6bd9dbbc76b4c6214348a32e4e72" Sep 16 14:00:04.697171 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.697148 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" Sep 16 14:00:04.697171 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.697159 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh" event={"ID":"52d4cce9-3deb-4de3-8c00-2d73a374a9c5","Type":"ContainerDied","Data":"fa6eb31e3ce4d073f7a6a45fac37f73f5e2c62ea57d829267aeb8adbf38e0629"} Sep 16 14:00:04.697415 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.697191 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa6eb31e3ce4d073f7a6a45fac37f73f5e2c62ea57d829267aeb8adbf38e0629" Sep 16 14:00:04.698652 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.698635 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" Sep 16 14:00:04.698761 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.698679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-qqs5t" event={"ID":"085b7f6b-5eb4-4354-978e-e13befdf67c8","Type":"ContainerDied","Data":"d8c814ed24836a812d85c9ba6008dba2e386034a9d50f34d901e942696b80e89"} Sep 16 14:00:04.698761 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:04.698722 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c814ed24836a812d85c9ba6008dba2e386034a9d50f34d901e942696b80e89" Sep 16 14:00:36.131666 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:00:36.131633 2581 scope.go:117] "RemoveContainer" containerID="315d6d1898fc1d62ca6a864444b3133f856656b19465a02c92fc8bcd66008e54" Sep 16 14:01:36.156320 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:01:36.156276 2581 scope.go:117] "RemoveContainer" containerID="6027086f4798b5979de1b024218ac93418828073d653771b36e32c706e7ed612" Sep 16 14:03:10.703444 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:10.703415 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kpbdv_c1439ca8-6576-4e08-8093-cea64ff568cb/server/0.log" Sep 16 14:03:10.815843 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:10.815817 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sqljq_cd31f8ec-13d8-4a5f-a96c-678b72e4d82b/manager/0.log" Sep 16 14:03:10.934395 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:10.934367 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-7x9xr_22ae90fe-7952-4ec2-9476-311dce8b45c5/copy-manifests/0.log" Sep 16 14:03:10.940825 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:10.940804 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-7x9xr_22ae90fe-7952-4ec2-9476-311dce8b45c5/manager/0.log" Sep 16 14:03:12.186893 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:12.186863 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826090-vljns_3fc99f3b-6a17-4f8d-8158-cd2c463031c6/cleanup/0.log" Sep 16 14:03:12.298792 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:12.298762 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826105-jhn6v_f0220603-ee86-41ed-8b34-b18887c8b211/cleanup/0.log" Sep 16 14:03:12.414271 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:12.414245 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826120-qqs5t_085b7f6b-5eb4-4354-978e-e13befdf67c8/cleanup/0.log" Sep 16 14:03:12.528015 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:12.527990 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-48adf5-29826090-fz5rs_22a4d87f-c263-4b9c-b18b-361a4dab14a7/cleanup/0.log" Sep 16 14:03:12.653710 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:12.653679 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-a-48adf5-29826105-xcqwm_3e38a71d-2a73-4aae-9d4f-04509f1f627f/cleanup/0.log" Sep 16 14:03:13.024095 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:13.024066 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-c71ad7-29826105-9j9tc_412fa616-75fa-488c-a971-b481a6e429b7/cleanup/0.log" Sep 16 14:03:13.133988 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:13.133961 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-inf-b-c71ad7-29826120-cwkk2_6f310810-88ab-4075-8eb7-f67f89d81696/cleanup/0.log" Sep 16 14:03:13.271145 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:13.271112 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w0-a21a40-29826090-7dpd7_1dfc26f0-2732-4a54-a254-0e75618cbf04/cleanup/0.log" Sep 16 14:03:13.386840 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:13.386757 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w0-a21a40-29826105-b5ftl_56fe3e98-996d-4bbd-a89e-1560d9372eea/cleanup/0.log" Sep 16 14:03:13.499862 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:13.499830 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w0-a21a40-29826120-5lzjh_52d4cce9-3deb-4de3-8c00-2d73a374a9c5/cleanup/0.log" Sep 16 14:03:13.618550 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:13.618517 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w4-f27ea8-29826090-fkst5_5aea6ca3-24ff-46eb-8947-e8827f0ca741/cleanup/0.log" Sep 16 14:03:15.332354 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:15.332329 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2l9p7_f93b15a2-5658-4cf3-b16c-6a4abfe07b44/manager/0.log" Sep 16 14:03:15.577590 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:15.577565 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-744dj_24efcd0c-c733-4f6b-bae7-e18ac2953f20/manager/0.log" Sep 16 14:03:15.962508 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:15.962482 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff_7790b7b1-ee07-4883-a478-3ed9c23a986e/istio-proxy/0.log" Sep 16 14:03:16.088097 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:16.088069 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-l45df_6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075/discovery/0.log" Sep 16 14:03:16.320590 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:16.320515 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6445dbdc5b-gkbm6_2585c164-4944-4ccc-98a7-750dcec12145/kube-auth-proxy/0.log" Sep 16 14:03:16.553492 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:16.553457 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-l46wn_8e4d6f37-8a71-4355-937a-be27626ab4d5/payload-pre-processing/0.log" Sep 16 14:03:16.688749 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:16.688673 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6_b8268d4f-2d67-4f3c-b757-61161d1c2b2d/payload-pre-processing/0.log" Sep 16 14:03:17.047641 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:17.047612 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2_4244d0df-31cb-4fe0-9953-8145ae2a3e88/payload-pre-processing/0.log" Sep 16 14:03:17.176413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:17.176380 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bdkgr_82ffd13e-bd27-4e53-b18f-ec8e8845f2a7/payload-processing/0.log" Sep 16 14:03:17.292709 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:17.292682 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd_238e1be0-8ff6-466e-9e94-68904cb8290a/payload-processing/0.log" Sep 16 14:03:17.523762 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:17.523731 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl_14c496d8-d00a-42b5-9e18-35d42904f774/payload-processing/0.log" Sep 16 14:03:17.643264 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:17.643231 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5_f809c5ee-091c-4d14-a5bd-a28baa1d5f41/payload-processing/0.log" Sep 16 14:03:18.277193 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:18.277169 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5lkfj_66dd56f5-81b4-4f2c-a4cc-e027b4f4e3a2/main/0.log" Sep 16 14:03:18.746647 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:18.746620 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f8xc7_46a1594a-f60f-4644-9a02-301a020c017a/main/0.log" Sep 16 14:03:26.801724 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:26.801675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:27.721480 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:27.721449 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zvh5v_b6dc3916-9abe-4021-a871-0deca7e008f1/global-pull-secret-syncer/0.log" Sep 16 14:03:27.777719 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:27.777695 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" Sep 16 14:03:27.868319 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:27.868295 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-73.ec2.internal_3467b75dabbe10a15451a6223ddb8b9c/haproxy/0.log" Sep 16 14:03:29.978628 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:29.978598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hpj8d_9ef40846-8a2e-4db0-9b5c-7a0f68d62b40/konnectivity-agent/0.log" limit=1024 Sep 16 14:03:32.270243 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:32.270214 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2l9p7_f93b15a2-5658-4cf3-b16c-6a4abfe07b44/manager/0.log" Sep 16 14:03:32.378761 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:32.378728 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-744dj_24efcd0c-c733-4f6b-bae7-e18ac2953f20/manager/0.log" Sep 16 14:03:34.409209 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:34.409179 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtm8g_4591d7af-e2b3-4d00-b38d-2797cfc3a086/node-exporter/0.log" Sep 16 14:03:34.436481 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:34.436457 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtm8g_4591d7af-e2b3-4d00-b38d-2797cfc3a086/kube-rbac-proxy/0.log" Sep 16 14:03:34.463984 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:34.463964 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dtm8g_4591d7af-e2b3-4d00-b38d-2797cfc3a086/init-textfile/0.log" Sep 16 14:03:35.203986 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:35.203958 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-gtrw4_e3d6c06b-3997-49de-baf6-6f7f1f8e8c85/prometheus-operator-admission-webhook/0.log" Sep 16 14:03:35.722413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:35.722384 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 14:03:35.740355 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:35.740331 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w7mng_0bfe9ac2-4ce9-4ff9-957c-a6fc6e308e81/ovn-acl-logging/0.log" Sep 16 14:03:36.874894 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:36.874864 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-h8ct8_e8435540-7eea-40c4-a6ef-9395e7986931/networking-console-plugin/0.log" Sep 16 14:03:38.025813 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.025776 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dbcf769b9-spqgv_a310f581-0a70-434f-a942-7aae07764397/console/0.log" Sep 16 14:03:38.100884 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.100852 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg"] Sep 16 14:03:38.101343 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101328 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f310810-88ab-4075-8eb7-f67f89d81696" containerName="cleanup" Sep 16 14:03:38.101413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101344 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f310810-88ab-4075-8eb7-f67f89d81696" containerName="cleanup" Sep 16 14:03:38.101413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101361 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b7f6b-5eb4-4354-978e-e13befdf67c8" containerName="cleanup" Sep 16 14:03:38.101413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101366 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b7f6b-5eb4-4354-978e-e13befdf67c8" containerName="cleanup" Sep 16 14:03:38.101413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101381 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d4cce9-3deb-4de3-8c00-2d73a374a9c5" containerName="cleanup" Sep 16 14:03:38.101413 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101388 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d4cce9-3deb-4de3-8c00-2d73a374a9c5" containerName="cleanup" Sep 16 14:03:38.101580 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101448 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f310810-88ab-4075-8eb7-f67f89d81696" containerName="cleanup" Sep 16 14:03:38.101580 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101455 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b7f6b-5eb4-4354-978e-e13befdf67c8" containerName="cleanup" Sep 16 14:03:38.101580 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.101461 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d4cce9-3deb-4de3-8c00-2d73a374a9c5" containerName="cleanup" Sep 16 14:03:38.104686 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.104667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.107581 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.107558 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hrntr\"/\"kube-root-ca.crt\"" Sep 16 14:03:38.107690 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.107667 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hrntr\"/\"openshift-service-ca.crt\"" Sep 16 14:03:38.108505 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.108490 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hrntr\"/\"default-dockercfg-pll6d\"" Sep 16 14:03:38.124110 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.124079 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg"] Sep 16 14:03:38.205363 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.205339 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qcv\" (UniqueName: \"kubernetes.io/projected/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-kube-api-access-v2qcv\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.205476 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.205385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-proc\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.205476 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.205442 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-podres\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.205476 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.205463 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-sys\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.205608 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.205480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-lib-modules\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306378 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-lib-modules\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306378 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qcv\" (UniqueName: \"kubernetes.io/projected/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-kube-api-access-v2qcv\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-proc\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306466 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-lib-modules\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-podres\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-sys\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-proc\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306566 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-podres\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.306785 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.306582 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-sys\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.314453 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.314423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qcv\" (UniqueName: \"kubernetes.io/projected/d8ffaa62-fea8-4875-ba6c-8efd35db8cec-kube-api-access-v2qcv\") pod \"perf-node-gather-daemonset-dvgqg\" (UID: \"d8ffaa62-fea8-4875-ba6c-8efd35db8cec\") " pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.414406 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.414380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.689739 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.689713 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg"] Sep 16 14:03:38.690417 ip-10-0-133-73 kubenswrapper[2581]: W0916 14:03:38.690394 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8ffaa62_fea8_4875_ba6c_8efd35db8cec.slice/crio-155a4fdb7b239a3121847f1f3443a1328a38bd19e52e25c3e548c476ccc1005a WatchSource:0}: Error finding container 155a4fdb7b239a3121847f1f3443a1328a38bd19e52e25c3e548c476ccc1005a: Status 404 returned error can't find the container with id 155a4fdb7b239a3121847f1f3443a1328a38bd19e52e25c3e548c476ccc1005a Sep 16 14:03:38.975033 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.974993 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" event={"ID":"d8ffaa62-fea8-4875-ba6c-8efd35db8cec","Type":"ContainerStarted","Data":"0d42e7c4bd41b0dd1648aece8c32c5372c5d50494243a6457cab3f2a52ab52b7"} Sep 16 14:03:38.975219 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.975067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" event={"ID":"d8ffaa62-fea8-4875-ba6c-8efd35db8cec","Type":"ContainerStarted","Data":"155a4fdb7b239a3121847f1f3443a1328a38bd19e52e25c3e548c476ccc1005a"} Sep 16 14:03:38.975219 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.975095 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:38.991543 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:38.991497 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" podStartSLOduration=0.991481621 podStartE2EDuration="991.481621ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:38.99012802 +0000 UTC m=+3304.109854606" watchObservedRunningTime="2026-09-16 14:03:38.991481621 +0000 UTC m=+3304.111208170" Sep 16 14:03:39.957186 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:39.957149 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmsv9_90fdd31a-1830-4d96-8a94-f3083f6d9b85/dns/0.log" Sep 16 14:03:39.980499 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:39.980477 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmsv9_90fdd31a-1830-4d96-8a94-f3083f6d9b85/kube-rbac-proxy/0.log" Sep 16 14:03:40.071715 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:40.071687 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m2lwv_d8ec3cf3-f7e9-416b-9a45-2faf2cd25ece/dns-node-resolver/0.log" Sep 16 14:03:40.712839 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:40.712803 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66f4746c99-qwlfr_dd7594e4-0006-4edd-8904-9ec96fa3ccaf/registry/0.log" Sep 16 14:03:40.748411 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:40.748371 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lv88k_00be8b34-1abf-4801-b955-ba483d5f6c34/node-ca/0.log" Sep 16 14:03:41.917815 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:41.917785 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff_7790b7b1-ee07-4883-a478-3ed9c23a986e/istio-proxy/0.log" Sep 16 14:03:42.281917 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.281886 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-l45df_6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075/discovery/0.log" Sep 16 14:03:42.338970 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.338942 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6445dbdc5b-gkbm6_2585c164-4944-4ccc-98a7-750dcec12145/kube-auth-proxy/0.log" Sep 16 14:03:42.397742 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.397716 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-l46wn_8e4d6f37-8a71-4355-937a-be27626ab4d5/payload-pre-processing/0.log" Sep 16 14:03:42.419861 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.419836 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6_b8268d4f-2d67-4f3c-b757-61161d1c2b2d/payload-pre-processing/0.log" Sep 16 14:03:42.505582 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.505548 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2_4244d0df-31cb-4fe0-9953-8145ae2a3e88/payload-pre-processing/0.log" Sep 16 14:03:42.539604 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.539529 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bdkgr_82ffd13e-bd27-4e53-b18f-ec8e8845f2a7/payload-processing/0.log" Sep 16 14:03:42.564578 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.564545 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd_238e1be0-8ff6-466e-9e94-68904cb8290a/payload-processing/0.log" Sep 16 14:03:42.618271 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.618249 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl_14c496d8-d00a-42b5-9e18-35d42904f774/payload-processing/0.log" Sep 16 14:03:42.648228 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:42.648194 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5_f809c5ee-091c-4d14-a5bd-a28baa1d5f41/payload-processing/0.log" Sep 16 14:03:43.600168 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:43.600141 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-24dv4_cd78deae-72ad-4d9c-a31d-e423437d8169/serve-healthcheck-canary/0.log" Sep 16 14:03:44.313088 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:44.313058 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzkfk_1a8564d5-0e0d-4262-8b90-3af68cb8574e/kube-rbac-proxy/0.log" Sep 16 14:03:44.337217 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:44.337188 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzkfk_1a8564d5-0e0d-4262-8b90-3af68cb8574e/exporter/0.log" Sep 16 14:03:44.360907 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:44.360885 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzkfk_1a8564d5-0e0d-4262-8b90-3af68cb8574e/extractor/0.log" Sep 16 14:03:44.991431 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:44.991396 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hrntr/perf-node-gather-daemonset-dvgqg" Sep 16 14:03:47.033234 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:47.033211 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kpbdv_c1439ca8-6576-4e08-8093-cea64ff568cb/server/0.log" Sep 16 14:03:47.137464 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:47.137435 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sqljq_cd31f8ec-13d8-4a5f-a96c-678b72e4d82b/manager/0.log" Sep 16 14:03:47.172390 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:47.172364 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-7x9xr_22ae90fe-7952-4ec2-9476-311dce8b45c5/manager/0.log" Sep 16 14:03:47.207859 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:47.207837 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694ff5d95c-7x9xr_22ae90fe-7952-4ec2-9476-311dce8b45c5/copy-manifests/0.log" Sep 16 14:03:47.879263 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:47.879188 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff_7790b7b1-ee07-4883-a478-3ed9c23a986e/istio-proxy/0.log" Sep 16 14:03:48.208965 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.208936 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-l45df_6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075/discovery/0.log" Sep 16 14:03:48.255453 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.255426 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6445dbdc5b-gkbm6_2585c164-4944-4ccc-98a7-750dcec12145/kube-auth-proxy/0.log" Sep 16 14:03:48.327909 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.327881 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-l46wn_8e4d6f37-8a71-4355-937a-be27626ab4d5/payload-pre-processing/0.log" Sep 16 14:03:48.355672 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.355643 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6_b8268d4f-2d67-4f3c-b757-61161d1c2b2d/payload-pre-processing/0.log" Sep 16 14:03:48.400580 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.400557 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2_4244d0df-31cb-4fe0-9953-8145ae2a3e88/payload-pre-processing/0.log" Sep 16 14:03:48.419978 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.419956 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bdkgr_82ffd13e-bd27-4e53-b18f-ec8e8845f2a7/payload-processing/0.log" Sep 16 14:03:48.431041 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.431023 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd_238e1be0-8ff6-466e-9e94-68904cb8290a/payload-processing/0.log" Sep 16 14:03:48.458029 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.458013 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl_14c496d8-d00a-42b5-9e18-35d42904f774/payload-processing/0.log" Sep 16 14:03:48.471187 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.471135 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5_f809c5ee-091c-4d14-a5bd-a28baa1d5f41/payload-processing/0.log" Sep 16 14:03:48.847780 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:48.847709 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-bfz7b_c9afd9e0-93e5-42f7-9a4e-5558e89fc6c1/openshift-lws-operator/0.log" Sep 16 14:03:49.969954 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:49.969927 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4qxff_7790b7b1-ee07-4883-a478-3ed9c23a986e/istio-proxy/0.log" Sep 16 14:03:50.300729 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.300695 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-l45df_6ec86c9b-8f0a-4cb6-9964-9eebe4fc0075/discovery/0.log" Sep 16 14:03:50.332341 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.332297 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6445dbdc5b-gkbm6_2585c164-4944-4ccc-98a7-750dcec12145/kube-auth-proxy/0.log" Sep 16 14:03:50.370721 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.370688 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-l46wn_8e4d6f37-8a71-4355-937a-be27626ab4d5/payload-pre-processing/0.log" Sep 16 14:03:50.385592 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.385552 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-48adf5-6f4db764db-v5tz6_b8268d4f-2d67-4f3c-b757-61161d1c2b2d/payload-pre-processing/0.log" Sep 16 14:03:50.432893 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.432868 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-f27ea8-6b5d6fcf94-f5cm2_4244d0df-31cb-4fe0-9953-8145ae2a3e88/payload-pre-processing/0.log" Sep 16 14:03:50.450041 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.450022 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bdkgr_82ffd13e-bd27-4e53-b18f-ec8e8845f2a7/payload-processing/0.log" Sep 16 14:03:50.461903 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.461857 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-48adf5-6b465d865c-fvrjd_238e1be0-8ff6-466e-9e94-68904cb8290a/payload-processing/0.log" Sep 16 14:03:50.492752 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.492730 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-a21a40-7d5576cd45-bqjwl_14c496d8-d00a-42b5-9e18-35d42904f774/payload-processing/0.log" Sep 16 14:03:50.506473 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:50.506454 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-f27ea8-5d69d75685-xgtc5_f809c5ee-091c-4d14-a5bd-a28baa1d5f41/payload-processing/0.log" Sep 16 14:03:54.072488 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:54.072458 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-fwkkc_41ad6f3d-f396-4a4c-8754-b2126090c34b/migrator/0.log" Sep 16 14:03:54.094657 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:54.094629 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-fwkkc_41ad6f3d-f396-4a4c-8754-b2126090c34b/graceful-termination/0.log" Sep 16 14:03:55.467034 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.466998 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5k5jw_1d00521c-a61f-45d7-8f29-68d7b45a82ea/kube-multus/0.log" Sep 16 14:03:55.496012 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.495992 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/kube-multus-additional-cni-plugins/0.log" Sep 16 14:03:55.520941 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.520923 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/egress-router-binary-copy/0.log" Sep 16 14:03:55.544841 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.544823 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/cni-plugins/0.log" Sep 16 14:03:55.566588 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.566567 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/bond-cni-plugin/0.log" Sep 16 14:03:55.592121 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.592102 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/routeoverride-cni/0.log" Sep 16 14:03:55.614882 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.614846 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/whereabouts-cni-bincopy/0.log" Sep 16 14:03:55.638582 ip-10-0-133-73 kubenswrapper[2581]: I0916 14:03:55.638553 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bmbbz_75bd51a9-a796-46f3-ae52-fb1b49353736/whereabouts-cni/0.log"