Aug 05 16:22:48.427364 ip-10-0-131-60 systemd[1]: Starting Kubernetes Kubelet... Aug 05 16:22:48.797636 ip-10-0-131-60 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:22:48.797636 ip-10-0-131-60 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 05 16:22:48.797636 ip-10-0-131-60 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:22:48.797636 ip-10-0-131-60 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 05 16:22:48.797636 ip-10-0-131-60 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:22:48.798376 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.798293 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 05 16:22:48.802724 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802708 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:48.802724 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802723 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802727 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802730 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802733 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802736 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802739 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802745 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802748 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802751 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802755 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802758 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802761 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802764 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802766 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802769 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802772 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802775 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802782 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802785 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:48.802790 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802788 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802790 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802793 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802796 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802798 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802801 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802805 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802808 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802811 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802815 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802818 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802821 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802823 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802826 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802829 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802832 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802834 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802837 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802839 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:48.803254 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802842 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802844 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802847 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802849 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802852 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802854 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802858 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802861 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802863 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802866 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802868 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802871 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802874 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802876 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802879 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802882 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802885 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802888 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802890 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802893 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:48.803719 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802895 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802898 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802900 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802903 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802906 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802908 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802911 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802913 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802916 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802918 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802921 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802923 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802926 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802929 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802931 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802934 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802936 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802939 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802942 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802944 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:48.804230 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802947 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802949 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802953 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802957 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802959 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802963 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802966 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802970 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802973 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802976 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802979 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802981 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802984 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802987 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802989 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802992 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802995 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.802997 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803000 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:48.805050 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803003 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803757 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803765 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803768 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803770 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803774 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803776 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803779 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803782 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803785 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803787 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803790 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803792 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803795 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803797 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803800 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803803 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803805 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803808 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803811 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:48.805570 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803814 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803817 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803819 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803822 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803825 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803827 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803830 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803833 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803836 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803840 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803844 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803847 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803850 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803853 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803856 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803859 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803861 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803864 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803867 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803869 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:48.806069 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803872 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803874 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803877 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803880 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803882 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803885 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803887 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803890 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803892 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803895 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803897 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803900 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803905 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803908 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803910 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803913 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803915 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803918 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803920 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803923 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:48.806581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803925 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803928 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803930 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803932 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803935 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803938 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803941 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803943 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803946 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803948 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803951 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803954 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803956 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803959 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803961 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803966 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803969 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803972 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803975 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:48.807066 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803979 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803981 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803984 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803987 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803989 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803992 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803995 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.803998 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804000 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804003 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804005 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804008 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804010 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804013 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804015 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804018 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804020 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804022 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804025 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:48.807651 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804027 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.804030 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805643 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805653 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805656 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805661 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805665 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805668 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805671 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805673 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805678 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805681 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805685 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805688 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805690 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805693 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805697 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805700 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805703 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:48.808137 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805706 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805708 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805711 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805714 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805717 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805720 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805722 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805725 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805727 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805730 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805732 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805735 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805738 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805741 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805744 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805747 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805749 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805752 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805755 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:48.808599 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805757 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805760 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805763 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805765 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805768 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805771 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805773 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805776 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805780 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805782 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805785 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805788 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805791 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805794 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805796 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805799 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805802 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805805 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805808 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805810 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:48.809055 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805813 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805816 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805819 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805821 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805824 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805826 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805829 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805831 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805834 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805837 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805839 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805842 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805844 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805847 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805850 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805852 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805855 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805858 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805861 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805863 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:48.809541 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805865 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805868 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805874 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805877 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805882 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805886 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805889 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805902 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805905 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805908 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805911 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805914 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805916 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805919 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805922 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805924 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805927 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805930 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805932 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:48.810051 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805935 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805937 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805940 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.805942 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806008 2567 flags.go:64] FLAG: --address="0.0.0.0" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806018 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806024 2567 flags.go:64] FLAG: --anonymous-auth="true" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806029 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806033 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806036 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806040 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806044 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806047 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806050 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806054 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806059 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806062 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806065 2567 flags.go:64] FLAG: --cgroup-root="" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806068 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806071 2567 flags.go:64] FLAG: --client-ca-file="" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806074 2567 flags.go:64] FLAG: --cloud-provider="external" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806098 2567 flags.go:64] FLAG: --cluster-dns="[]" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806106 2567 flags.go:64] FLAG: --cluster-domain="" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806111 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 05 16:22:48.810532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806115 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806118 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806121 2567 flags.go:64] FLAG: --container-log-max-files="5" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806125 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806128 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806131 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806134 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806137 2567 flags.go:64] FLAG: --contention-profiling="false" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806140 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806143 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806146 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806149 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806153 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806156 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806159 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806161 2567 flags.go:64] FLAG: --enable-load-reader="false" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806165 2567 flags.go:64] FLAG: --enable-server="true" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806167 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806172 2567 flags.go:64] FLAG: --event-burst="100" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806175 2567 flags.go:64] FLAG: --event-qps="50" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806178 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806181 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806185 2567 flags.go:64] FLAG: --eviction-hard="" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806190 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 05 16:22:48.811083 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806193 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806196 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806199 2567 flags.go:64] FLAG: --eviction-soft="" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806202 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806204 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806207 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806217 2567 flags.go:64] FLAG: --experimental-mounter-path="" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806220 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806223 2567 flags.go:64] FLAG: --fail-swap-on="true" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806225 2567 flags.go:64] FLAG: --feature-gates="" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806229 2567 flags.go:64] FLAG: --file-check-frequency="20s" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806232 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806235 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806238 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806242 2567 flags.go:64] FLAG: --healthz-port="10248" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806244 2567 flags.go:64] FLAG: --help="false" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806247 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-131-60.ec2.internal" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806250 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806253 2567 flags.go:64] FLAG: --http-check-frequency="20s" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806256 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806259 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806263 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806265 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 05 16:22:48.811703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806268 2567 flags.go:64] FLAG: --image-service-endpoint="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806271 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806273 2567 flags.go:64] FLAG: --kube-api-burst="100" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806276 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806279 2567 flags.go:64] FLAG: --kube-api-qps="50" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806282 2567 flags.go:64] FLAG: --kube-reserved="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806285 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806288 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806293 2567 flags.go:64] FLAG: --kubelet-cgroups="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806295 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806298 2567 flags.go:64] FLAG: --lock-file="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806301 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806304 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806307 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806312 2567 flags.go:64] FLAG: --log-json-split-stream="false" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806315 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806323 2567 flags.go:64] FLAG: --log-text-split-stream="false" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806326 2567 flags.go:64] FLAG: --logging-format="text" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806329 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806332 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806335 2567 flags.go:64] FLAG: --manifest-url="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806338 2567 flags.go:64] FLAG: --manifest-url-header="" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806342 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806346 2567 flags.go:64] FLAG: --max-open-files="1000000" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806350 2567 flags.go:64] FLAG: --max-pods="110" Aug 05 16:22:48.812295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806352 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806356 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806358 2567 flags.go:64] FLAG: --memory-manager-policy="None" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806361 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806364 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806367 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806370 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806377 2567 flags.go:64] FLAG: --node-status-max-images="50" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806381 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806384 2567 flags.go:64] FLAG: --oom-score-adj="-999" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806386 2567 flags.go:64] FLAG: --pod-cidr="" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806389 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806394 2567 flags.go:64] FLAG: --pod-manifest-path="" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806397 2567 flags.go:64] FLAG: --pod-max-pids="-1" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806402 2567 flags.go:64] FLAG: --pods-per-core="0" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806404 2567 flags.go:64] FLAG: --port="10250" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806409 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806412 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1a49e2d2b00a60d" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806415 2567 flags.go:64] FLAG: --qos-reserved="" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806418 2567 flags.go:64] FLAG: --read-only-port="10255" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806421 2567 flags.go:64] FLAG: --register-node="true" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806424 2567 flags.go:64] FLAG: --register-with-taints="" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806427 2567 flags.go:64] FLAG: --registry-burst="10" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806430 2567 flags.go:64] FLAG: --registry-qps="5" Aug 05 16:22:48.813206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806439 2567 flags.go:64] FLAG: --reserved-cpus="" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806442 2567 flags.go:64] FLAG: --reserved-memory="" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806445 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806449 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806451 2567 flags.go:64] FLAG: --rotate-certificates="false" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806454 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806457 2567 flags.go:64] FLAG: --runonce="false" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806460 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806463 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806466 2567 flags.go:64] FLAG: --seccomp-default="false" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806469 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806472 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806475 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806478 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806481 2567 flags.go:64] FLAG: --storage-driver-password="root" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806484 2567 flags.go:64] FLAG: --storage-driver-secure="false" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806487 2567 flags.go:64] FLAG: --storage-driver-table="stats" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806490 2567 flags.go:64] FLAG: --storage-driver-user="root" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806492 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806495 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806498 2567 flags.go:64] FLAG: --system-cgroups="" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806501 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806508 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806511 2567 flags.go:64] FLAG: --tls-cert-file="" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806514 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 05 16:22:48.814285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806521 2567 flags.go:64] FLAG: --tls-min-version="" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806524 2567 flags.go:64] FLAG: --tls-private-key-file="" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806527 2567 flags.go:64] FLAG: --topology-manager-policy="none" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806530 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806533 2567 flags.go:64] FLAG: --topology-manager-scope="container" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806536 2567 flags.go:64] FLAG: --v="2" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806539 2567 flags.go:64] FLAG: --version="false" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806543 2567 flags.go:64] FLAG: --vmodule="" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806552 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.806556 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806662 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806666 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806670 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806674 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806677 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806680 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806683 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806685 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806688 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806690 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806693 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:48.815403 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806695 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806698 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806701 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806703 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806706 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806708 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806710 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806713 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806717 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806719 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806723 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806725 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806728 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806730 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806733 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806736 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806738 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806741 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806743 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:48.816327 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806746 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806755 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806758 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806761 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806763 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806765 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806770 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806773 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806776 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806778 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806782 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806784 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806787 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806790 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806792 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806795 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806798 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806800 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806803 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:48.816858 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806805 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806808 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806812 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806814 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806817 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806820 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806822 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806825 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806827 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806830 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806832 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806835 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806837 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806840 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806842 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806850 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806853 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806855 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806858 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806860 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:48.817661 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806863 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806866 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806868 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806871 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806873 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806876 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806878 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806881 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806883 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806886 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806888 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806891 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806894 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806897 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806901 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806904 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806907 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806910 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806912 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806915 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:48.818436 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806918 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806920 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806922 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806925 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806928 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806930 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806933 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806935 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806944 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.806947 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.807456 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.814809 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.814829 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.814854 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.814864 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 05 16:22:48.819003 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814946 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814953 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814958 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814963 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814968 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814973 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814978 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814983 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814988 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814992 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.814997 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815001 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815008 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815014 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815018 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815023 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815027 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815032 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815036 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815041 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:48.819424 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815045 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815049 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815054 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815058 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815063 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815068 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815073 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815077 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815082 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815103 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815107 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815111 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815115 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815118 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815122 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815126 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815130 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815133 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815137 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:48.819933 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815142 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815148 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815154 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815159 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815163 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815169 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815173 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815178 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815182 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815186 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815191 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815195 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815200 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815204 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815209 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815213 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815218 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815222 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815227 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815234 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:48.820455 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815238 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815243 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815248 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815253 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815257 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815261 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815265 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815270 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815274 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815278 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815283 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815287 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815292 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815296 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815301 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815305 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815309 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815313 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815317 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815322 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:48.820942 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815326 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815330 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815335 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815339 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815344 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815348 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815352 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815356 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815360 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815364 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815369 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815373 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815379 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815384 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815388 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815392 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815397 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815401 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815407 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:48.821700 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815412 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.815420 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815602 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815611 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815618 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815623 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815628 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815633 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815637 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815642 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815646 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815651 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815655 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815659 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815663 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:22:48.822226 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815668 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815672 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815676 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815680 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815684 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815688 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815692 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815704 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815708 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815713 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815717 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815723 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815727 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815731 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815737 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815744 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815750 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815754 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815758 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:22:48.822848 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815763 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815767 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815772 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815776 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815780 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815784 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815788 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815792 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815796 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815800 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815805 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815809 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815813 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815817 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815821 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815825 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815829 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815833 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815837 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815841 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:22:48.823620 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815845 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815850 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815854 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815858 2567 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815862 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815868 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815872 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815877 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815881 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815885 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815889 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815893 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815897 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815901 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815905 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815909 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815913 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815917 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815921 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815925 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:22:48.824467 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815930 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815934 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815941 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815946 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815950 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815955 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815959 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815964 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815968 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815972 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815976 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815980 2567 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815984 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815988 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815992 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.815997 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816001 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816005 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816011 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:22:48.825018 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816015 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816020 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816024 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816028 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816032 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816036 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816040 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:48.816044 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.816053 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.816213 2567 server.go:962] "Client rotation is on, will bootstrap in background" Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.818878 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.820323 2567 server.go:1019] "Starting client certificate rotation" Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.820432 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 16:22:48.825522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.820463 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 16:22:48.825871 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.822759 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 16:22:48.825871 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.824585 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 16:22:48.836120 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.836099 2567 log.go:25] "Validated CRI v1 runtime API" Aug 05 16:22:48.838790 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.838774 2567 log.go:25] "Validated CRI v1 image API" Aug 05 16:22:48.839778 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.839749 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 05 16:22:48.842676 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.842654 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f4f856bf-b710-4bd4-aec3-9e6517200566:/dev/nvme0n1p4 f9f6d43e-e97a-4560-a1d3-658be063741d:/dev/nvme0n1p3] Aug 05 16:22:48.842725 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.842676 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 05 16:22:48.849640 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.849521 2567 manager.go:216] Machine: {Timestamp:2026-08-05 16:22:48.848291933 +0000 UTC m=+0.308094205 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100936 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd4e04e43d37c97c78e1a9410b938 SystemUUID:ec2bd4e0-4e43-d37c-97c7-8e1a9410b938 BootID:3b05ffa2-ae8d-493f-80b5-9664c5fcb81e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cf:7b:62:65:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cf:7b:62:65:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:3f:54:77:96:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 05 16:22:48.849640 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.849631 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 05 16:22:48.849781 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.849714 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 05 16:22:48.850947 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.850929 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 16:22:48.852628 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.852595 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 05 16:22:48.852792 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.852633 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-60.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 05 16:22:48.853759 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.853745 2567 topology_manager.go:138] "Creating topology manager with none policy" Aug 05 16:22:48.853823 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.853763 2567 container_manager_linux.go:309] "Creating device plugin manager" Aug 05 16:22:48.853823 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.853775 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 16:22:48.854337 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.854324 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 16:22:48.855103 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.855076 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 05 16:22:48.858206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858192 2567 state_mem.go:36] "Initialized new in-memory state store" Aug 05 16:22:48.858339 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858328 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 05 16:22:48.858406 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858367 2567 kubelet.go:486] "Attempting to sync node with API server" Aug 05 16:22:48.858406 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858381 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 05 16:22:48.858406 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858401 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 05 16:22:48.858532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858415 2567 kubelet.go:398] "Adding apiserver pod source" Aug 05 16:22:48.858532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858426 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 05 16:22:48.858800 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858786 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 16:22:48.858876 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.858810 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 16:22:48.860760 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.860740 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 05 16:22:48.862073 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.862055 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 05 16:22:48.862153 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.862105 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 05 16:22:48.863264 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863247 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 05 16:22:48.863264 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863264 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863270 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863275 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863285 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863291 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863299 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863304 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863311 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863316 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863324 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 05 16:22:48.863356 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863329 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 05 16:22:48.863983 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863974 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 05 16:22:48.863983 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.863982 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 05 16:22:48.867435 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.867422 2567 server.go:1268] "Started kubelet" Aug 05 16:22:48.867541 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.867504 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 05 16:22:48.867630 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.867566 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 05 16:22:48.867662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.867654 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 05 16:22:48.868562 ip-10-0-131-60 systemd[1]: Started Kubernetes Kubelet. Aug 05 16:22:48.868809 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.868646 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 05 16:22:48.868990 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.868963 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-60.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 05 16:22:48.869067 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.868989 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 05 16:22:48.869350 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.869069 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-60.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 05 16:22:48.869861 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.869791 2567 server.go:310] "Adding debug handlers to kubelet server" Aug 05 16:22:48.873530 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.873493 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 05 16:22:48.873530 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.873519 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 05 16:22:48.873912 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.873892 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:48.874213 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.874187 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 05 16:22:48.874406 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.874389 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 05 16:22:48.874448 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.874410 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 05 16:22:48.874557 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.874389 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 05 16:22:48.874586 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.874548 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqrtg" Aug 05 16:22:48.874667 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.874655 2567 reconstruct.go:97] "Volume reconstruction finished" Aug 05 16:22:48.874731 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.874667 2567 reconciler.go:29] "Reconciler: start to sync state" Aug 05 16:22:48.874984 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.874120 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-60.ec2.internal.18c8f54e4b06cee1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-60.ec2.internal,UID:ip-10-0-131-60.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-60.ec2.internal,},FirstTimestamp:2026-08-05 16:22:48.867393249 +0000 UTC m=+0.327195521,LastTimestamp:2026-08-05 16:22:48.867393249 +0000 UTC m=+0.327195521,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-60.ec2.internal,}" Aug 05 16:22:48.875466 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875449 2567 factory.go:153] Registering CRI-O factory Aug 05 16:22:48.875547 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875470 2567 factory.go:223] Registration of the crio container factory successfully Aug 05 16:22:48.875547 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875534 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 05 16:22:48.875547 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875541 2567 factory.go:55] Registering systemd factory Aug 05 16:22:48.875547 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875547 2567 factory.go:223] Registration of the systemd container factory successfully Aug 05 16:22:48.875656 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875563 2567 factory.go:103] Registering Raw factory Aug 05 16:22:48.875656 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875575 2567 manager.go:1196] Started watching for new ooms in manager Aug 05 16:22:48.875985 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.875975 2567 manager.go:349] Starting recovery of all containers Aug 05 16:22:48.879465 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.879440 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqrtg" Aug 05 16:22:48.880814 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.880785 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-60.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 05 16:22:48.880939 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.880912 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 05 16:22:48.888532 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.888515 2567 manager.go:354] Recovery completed Aug 05 16:22:48.892591 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.892576 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:48.894268 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894246 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:48.894331 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894273 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:48.894331 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894283 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:48.894779 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894763 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 05 16:22:48.894779 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894775 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:48.894837 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894778 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 05 16:22:48.894837 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.894794 2567 state_mem.go:36] "Initialized new in-memory state store" Aug 05 16:22:48.895531 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.895514 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:48.895590 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.895545 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:48.895590 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.895558 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:48.895810 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.895794 2567 policy_none.go:49] "None policy: Start" Aug 05 16:22:48.895810 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.895811 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 05 16:22:48.895911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.895822 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 05 16:22:48.896685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.896676 2567 policy_none.go:47] "Start" Aug 05 16:22:48.932804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.932619 2567 manager.go:337] "Starting Device Plugin manager" Aug 05 16:22:48.932902 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.932835 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 05 16:22:48.932902 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.932851 2567 server.go:87] "Starting device plugin registration server" Aug 05 16:22:48.933059 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.933048 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 05 16:22:48.933112 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.933061 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 05 16:22:48.933715 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.933688 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 05 16:22:48.933806 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.933794 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 05 16:22:48.933806 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:48.933804 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 05 16:22:48.933897 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.933882 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 05 16:22:48.933946 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:48.933936 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.003869 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.003827 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 05 16:22:49.005804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.005053 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 05 16:22:49.005804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.005113 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 05 16:22:49.005804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.005149 2567 kubelet.go:2448] "Starting kubelet main sync loop" Aug 05 16:22:49.005804 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.005192 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 05 16:22:49.007668 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.007644 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:49.033847 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.033829 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:49.034654 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.034639 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:49.034718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.034669 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:49.034718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.034678 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:49.034718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.034701 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.040951 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.040936 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.041028 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.040963 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-60.ec2.internal\": node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.052114 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.052048 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.106130 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.106082 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal"] Aug 05 16:22:49.106242 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.106188 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:49.107016 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.107001 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:49.107082 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.107029 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:49.107082 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.107040 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:49.109188 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109176 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:49.109292 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.109336 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109309 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:49.109882 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109864 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:49.109936 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109897 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:49.109936 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109910 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:49.109995 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109872 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:49.109995 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109969 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:49.110054 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.109999 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:49.111965 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.111947 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.112041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.111997 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:22:49.112612 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.112598 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:22:49.112666 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.112624 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:22:49.112666 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.112633 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:22:49.133747 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.133726 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-60.ec2.internal\" not found" node="ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.138053 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.138038 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-60.ec2.internal\" not found" node="ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.153139 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.153121 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.176281 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.176262 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6718bd51b27ca92571804b6ccc53294d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal\" (UID: \"6718bd51b27ca92571804b6ccc53294d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.176363 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.176289 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6718bd51b27ca92571804b6ccc53294d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal\" (UID: \"6718bd51b27ca92571804b6ccc53294d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.176363 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.176304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39a7c3613bf8e5e7fd5e0d2f670dcc7b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.ec2.internal\" (UID: \"39a7c3613bf8e5e7fd5e0d2f670dcc7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.254180 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.254146 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.277269 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.277233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6718bd51b27ca92571804b6ccc53294d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal\" (UID: \"6718bd51b27ca92571804b6ccc53294d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.277269 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.277282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6718bd51b27ca92571804b6ccc53294d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal\" (UID: \"6718bd51b27ca92571804b6ccc53294d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.277472 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.277307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39a7c3613bf8e5e7fd5e0d2f670dcc7b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.ec2.internal\" (UID: \"39a7c3613bf8e5e7fd5e0d2f670dcc7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.277472 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.277306 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6718bd51b27ca92571804b6ccc53294d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal\" (UID: \"6718bd51b27ca92571804b6ccc53294d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.277472 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.277358 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6718bd51b27ca92571804b6ccc53294d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal\" (UID: \"6718bd51b27ca92571804b6ccc53294d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.277472 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.277366 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39a7c3613bf8e5e7fd5e0d2f670dcc7b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-60.ec2.internal\" (UID: \"39a7c3613bf8e5e7fd5e0d2f670dcc7b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.355829 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.355791 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.439796 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.439757 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.442129 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.442108 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.456235 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.456214 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.561160 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.561121 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.662341 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.662262 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-60.ec2.internal\" not found" Aug 05 16:22:49.682037 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.682010 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:49.775367 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.775341 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.787818 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.787794 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 16:22:49.789653 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.789637 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" Aug 05 16:22:49.798997 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.798980 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 16:22:49.820098 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.820072 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 05 16:22:49.820247 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.820227 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 16:22:49.820309 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.820276 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 16:22:49.858630 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.858604 2567 apiserver.go:52] "Watching apiserver" Aug 05 16:22:49.873807 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.873782 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 05 16:22:49.875579 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.875558 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 05 16:22:49.876046 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.876025 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:49.876243 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.876221 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kjwrw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5","openshift-cluster-node-tuning-operator/tuned-s5gqv","openshift-dns/node-resolver-zrpwk","openshift-image-registry/node-ca-bmlcr","openshift-multus/multus-additional-cni-plugins-prprc","openshift-multus/network-metrics-daemon-qs6sj","openshift-ovn-kubernetes/ovnkube-node-8hsb7","kube-system/konnectivity-agent-5ckqd","kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal","openshift-multus/multus-kwxld","openshift-network-diagnostics/network-check-target-ctr2g","openshift-network-operator/iptables-alerter-npp7d"] Aug 05 16:22:49.879081 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.879061 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:49.879200 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.879165 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:22:49.881784 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.881765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:49.881896 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.881835 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:22:49.883942 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.883911 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 16:17:49 +0000 UTC" deadline="2028-04-19 19:53:56.567595362 +0000 UTC" Aug 05 16:22:49.884056 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.883941 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14955h31m6.683658339s" Aug 05 16:22:49.884103 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.884073 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 16:22:49.884711 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.884696 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.886276 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.886259 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.886395 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.886258 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4j4jf\"" Aug 05 16:22:49.886395 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.886265 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.887862 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.887846 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:49.889372 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.889351 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.889461 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.889446 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5wqvc\"" Aug 05 16:22:49.889565 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.889551 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.890233 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.890218 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:49.891933 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.891915 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvj9p\"" Aug 05 16:22:49.892016 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.891917 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 05 16:22:49.892016 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.891950 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.892100 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.892029 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.893167 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.893147 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.894874 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.894856 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bd2tf\"" Aug 05 16:22:49.894954 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.894863 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 05 16:22:49.895018 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.894987 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 05 16:22:49.895070 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.895055 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.895125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.895102 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 05 16:22:49.895207 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.895157 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.895620 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.895605 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:49.897294 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.897276 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 05 16:22:49.897393 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.897379 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 05 16:22:49.897494 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.897478 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vp2kf\"" Aug 05 16:22:49.897834 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.897821 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:49.897890 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:49.897875 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:22:49.901530 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.901290 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.903007 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.902906 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.903007 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.902907 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 05 16:22:49.903220 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.903202 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n5jnv\"" Aug 05 16:22:49.903287 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.903218 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.904667 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.904254 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:49.905814 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.905751 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.906233 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.906137 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sxx66\"" Aug 05 16:22:49.906233 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.906144 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.906233 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.906177 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 05 16:22:49.906503 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.906487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.907795 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.907774 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-75s99" Aug 05 16:22:49.908457 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908223 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 05 16:22:49.908457 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x92ct\"" Aug 05 16:22:49.908457 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908365 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 05 16:22:49.908689 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908518 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 05 16:22:49.908770 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908753 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 05 16:22:49.908844 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908827 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 05 16:22:49.908891 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.908870 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 05 16:22:49.909188 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.909168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.912389 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.912334 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 05 16:22:49.912535 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.912409 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcv72\"" Aug 05 16:22:49.915948 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.915928 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-75s99" Aug 05 16:22:49.976017 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.975996 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 05 16:22:49.981586 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-k8s-cni-cncf-io\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.981709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676a50f-361b-49f4-b68b-33549816ce4a-tmp-dir\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:49.981709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981630 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5bh\" (UniqueName: \"kubernetes.io/projected/6676a50f-361b-49f4-b68b-33549816ce4a-kube-api-access-2j5bh\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:49.981709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtf5\" (UniqueName: \"kubernetes.io/projected/cc776abf-676f-43e7-b4b5-7c68a29a7db7-kube-api-access-xrtf5\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:49.981837 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-os-release\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.981837 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-cni-bin\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.981920 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981840 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.981920 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-etc-kubernetes\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.981920 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981912 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-kubernetes\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e464b10-9c26-458f-a3e6-5effcba181f5-kubelet-config\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:49.982060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.981983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-tuned\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-cnibin\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.982060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982035 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a8caac-3a10-4067-acee-50b4d7bccf8e-agent-certs\") pod \"konnectivity-agent-5ckqd\" (UID: \"18a8caac-3a10-4067-acee-50b4d7bccf8e\") " pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-systemd\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982126 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-log-socket\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-cni-multus\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-kubelet\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57tcl\" (UniqueName: \"kubernetes.io/projected/3d9ade42-fe48-40ae-b8ec-7dc429172620-kube-api-access-57tcl\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysctl-conf\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982237 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad3f059-4b5e-4e0a-86ff-085088e730c0-iptables-alerter-script\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982262 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-var-lib-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982307 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982343 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc776abf-676f-43e7-b4b5-7c68a29a7db7-serviceca\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982404 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e464b10-9c26-458f-a3e6-5effcba181f5-dbus\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qzp\" (UniqueName: \"kubernetes.io/projected/691952e9-2ab0-4fce-be21-cf7bfe4be086-kube-api-access-85qzp\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982461 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-slash\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-hostroot\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.982504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwt8\" (UniqueName: \"kubernetes.io/projected/bad3f059-4b5e-4e0a-86ff-085088e730c0-kube-api-access-xwwt8\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmnm\" (UniqueName: \"kubernetes.io/projected/e965746d-d914-436f-b115-0d3ebcc71e6d-kube-api-access-ppmnm\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-modprobe-d\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-sys\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-lib-modules\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982611 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cnibin\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovnkube-config\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-conf-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:49.982802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982781 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6676a50f-361b-49f4-b68b-33549816ce4a-hosts-file\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-device-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982873 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfvh\" (UniqueName: \"kubernetes.io/projected/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-kube-api-access-mgfvh\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982897 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-cni-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-node-log\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982947 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-daemon-config\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a8caac-3a10-4067-acee-50b4d7bccf8e-konnectivity-ca\") pod \"konnectivity-agent-5ckqd\" (UID: \"18a8caac-3a10-4067-acee-50b4d7bccf8e\") " pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.982997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysconfig\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysctl-d\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/691952e9-2ab0-4fce-be21-cf7bfe4be086-tmp\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-systemd-units\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-systemd\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovn-node-metrics-cert\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-system-cni-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d9ade42-fe48-40ae-b8ec-7dc429172620-cni-binary-copy\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983266 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-var-lib-kubelet\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-os-release\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983337 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-cni-bin\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983382 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-host\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-run-netns\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983513 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksgj\" (UniqueName: \"kubernetes.io/projected/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-kube-api-access-kksgj\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-ovn\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-env-overrides\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983579 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-multus-certs\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpp9v\" (UniqueName: \"kubernetes.io/projected/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-kube-api-access-zpp9v\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.983635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983619 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-kubelet\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983637 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc776abf-676f-43e7-b4b5-7c68a29a7db7-host\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983653 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-system-cni-dir\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983673 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-socket-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad3f059-4b5e-4e0a-86ff-085088e730c0-host-slash\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-etc-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983788 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovnkube-script-lib\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983809 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-socket-dir-parent\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-netns\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-registration-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-sys-fs\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-run\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:49.984414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:49.983917 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-cni-netd\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.019146 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.019084 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6718bd51b27ca92571804b6ccc53294d.slice/crio-ea9f47ce3b0e3b9ed3ca13a056766519579ca711ea5dcaf7dc45caf7d17a2692 WatchSource:0}: Error finding container ea9f47ce3b0e3b9ed3ca13a056766519579ca711ea5dcaf7dc45caf7d17a2692: Status 404 returned error can't find the container with id ea9f47ce3b0e3b9ed3ca13a056766519579ca711ea5dcaf7dc45caf7d17a2692 Aug 05 16:22:50.021840 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.021816 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a7c3613bf8e5e7fd5e0d2f670dcc7b.slice/crio-d4548cd1ed38dbc14a6d7137bdc97861d75aa78b18a538c91a6842a6398d126e WatchSource:0}: Error finding container d4548cd1ed38dbc14a6d7137bdc97861d75aa78b18a538c91a6842a6398d126e: Status 404 returned error can't find the container with id d4548cd1ed38dbc14a6d7137bdc97861d75aa78b18a538c91a6842a6398d126e Aug 05 16:22:50.023872 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.023853 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:22:50.084935 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.084904 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-run-netns\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.084935 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.084937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.085066 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.084954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksgj\" (UniqueName: \"kubernetes.io/projected/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-kube-api-access-kksgj\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.085066 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.084979 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-ovn\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.085066 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-run-netns\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.085066 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-env-overrides\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.085066 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-multus-certs\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpp9v\" (UniqueName: \"kubernetes.io/projected/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-kube-api-access-zpp9v\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085023 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-ovn\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-kubelet\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc776abf-676f-43e7-b4b5-7c68a29a7db7-host\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-multus-certs\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085201 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc776abf-676f-43e7-b4b5-7c68a29a7db7-host\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-system-cni-dir\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085275 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-socket-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-system-cni-dir\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad3f059-4b5e-4e0a-86ff-085088e730c0-host-slash\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad3f059-4b5e-4e0a-86ff-085088e730c0-host-slash\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.085361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-etc-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovnkube-script-lib\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085363 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-socket-dir-parent\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-netns\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-socket-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-registration-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-etc-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-sys-fs\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-kubelet\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-run\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085530 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-registration-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-socket-dir-parent\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-netns\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-cni-netd\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-sys-fs\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-k8s-cni-cncf-io\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-run\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-cni-netd\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676a50f-361b-49f4-b68b-33549816ce4a-tmp-dir\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085621 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-run-k8s-cni-cncf-io\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5bh\" (UniqueName: \"kubernetes.io/projected/6676a50f-361b-49f4-b68b-33549816ce4a-kube-api-access-2j5bh\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085667 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtf5\" (UniqueName: \"kubernetes.io/projected/cc776abf-676f-43e7-b4b5-7c68a29a7db7-kube-api-access-xrtf5\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-os-release\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085695 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-env-overrides\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-cni-bin\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-etc-kubernetes\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-os-release\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085808 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-kubernetes\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-cni-bin\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-kubernetes\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.086883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e464b10-9c26-458f-a3e6-5effcba181f5-kubelet-config\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085881 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676a50f-361b-49f4-b68b-33549816ce4a-tmp-dir\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-etc-kubernetes\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-tuned\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e464b10-9c26-458f-a3e6-5effcba181f5-kubelet-config\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-cnibin\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a8caac-3a10-4067-acee-50b4d7bccf8e-agent-certs\") pod \"konnectivity-agent-5ckqd\" (UID: \"18a8caac-3a10-4067-acee-50b4d7bccf8e\") " pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-cnibin\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.085975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-systemd\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086027 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-systemd\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-log-socket\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086057 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-log-socket\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086068 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-cni-multus\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-kubelet\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-cni-multus\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57tcl\" (UniqueName: \"kubernetes.io/projected/3d9ade42-fe48-40ae-b8ec-7dc429172620-kube-api-access-57tcl\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086169 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-kubelet\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086181 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysctl-conf\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.087750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086189 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad3f059-4b5e-4e0a-86ff-085088e730c0-iptables-alerter-script\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-var-lib-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc776abf-676f-43e7-b4b5-7c68a29a7db7-serviceca\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysctl-conf\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-var-lib-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e464b10-9c26-458f-a3e6-5effcba181f5-dbus\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qzp\" (UniqueName: \"kubernetes.io/projected/691952e9-2ab0-4fce-be21-cf7bfe4be086-kube-api-access-85qzp\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086380 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-slash\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-hostroot\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086432 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e464b10-9c26-458f-a3e6-5effcba181f5-dbus\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086474 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-hostroot\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086433 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwt8\" (UniqueName: \"kubernetes.io/projected/bad3f059-4b5e-4e0a-86ff-085088e730c0-kube-api-access-xwwt8\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmnm\" (UniqueName: \"kubernetes.io/projected/e965746d-d914-436f-b115-0d3ebcc71e6d-kube-api-access-ppmnm\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.088685 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-modprobe-d\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-sys\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-slash\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086626 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-lib-modules\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-openvswitch\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086652 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cnibin\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovnkube-config\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-conf-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-etc-selinux\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086765 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad3f059-4b5e-4e0a-86ff-085088e730c0-iptables-alerter-script\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6676a50f-361b-49f4-b68b-33549816ce4a-hosts-file\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086792 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086816 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-lib-modules\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086827 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-modprobe-d\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc776abf-676f-43e7-b4b5-7c68a29a7db7-serviceca\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.089599 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-sys\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-cnibin\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086859 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6676a50f-361b-49f4-b68b-33549816ce4a-hosts-file\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086868 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-host-run-ovn-kubernetes\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086883 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-conf-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.086943 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-device-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.086995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovnkube-script-lib\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.087034 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:50.586993593 +0000 UTC m=+2.046795856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087045 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-device-dir\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfvh\" (UniqueName: \"kubernetes.io/projected/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-kube-api-access-mgfvh\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-cni-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-node-log\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-cni-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-daemon-config\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-node-log\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a8caac-3a10-4067-acee-50b4d7bccf8e-konnectivity-ca\") pod \"konnectivity-agent-5ckqd\" (UID: \"18a8caac-3a10-4067-acee-50b4d7bccf8e\") " pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysconfig\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysctl-d\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovnkube-config\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087411 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysconfig\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/691952e9-2ab0-4fce-be21-cf7bfe4be086-tmp\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-systemd-units\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087503 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-sysctl-d\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-systemd\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovn-node-metrics-cert\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-system-cni-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d9ade42-fe48-40ae-b8ec-7dc429172620-cni-binary-copy\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-var-lib-kubelet\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087761 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-os-release\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d9ade42-fe48-40ae-b8ec-7dc429172620-multus-daemon-config\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.090718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-cni-bin\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087861 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a8caac-3a10-4067-acee-50b4d7bccf8e-konnectivity-ca\") pod \"konnectivity-agent-5ckqd\" (UID: \"18a8caac-3a10-4067-acee-50b4d7bccf8e\") " pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-host\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-host-var-lib-cni-bin\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-host\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-os-release\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.088006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/691952e9-2ab0-4fce-be21-cf7bfe4be086-var-lib-kubelet\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087551 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-systemd-units\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.087585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-run-systemd\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.088058 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.088135 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:22:50.588117452 +0000 UTC m=+2.047919727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.088138 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d9ade42-fe48-40ae-b8ec-7dc429172620-system-cni-dir\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.088303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.088363 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d9ade42-fe48-40ae-b8ec-7dc429172620-cni-binary-copy\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.088458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.089777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/691952e9-2ab0-4fce-be21-cf7bfe4be086-etc-tuned\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.091223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.089961 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/691952e9-2ab0-4fce-be21-cf7bfe4be086-tmp\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.091757 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.090167 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-ovn-node-metrics-cert\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.091757 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.090185 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a8caac-3a10-4067-acee-50b4d7bccf8e-agent-certs\") pod \"konnectivity-agent-5ckqd\" (UID: \"18a8caac-3a10-4067-acee-50b4d7bccf8e\") " pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:50.096002 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.095986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtf5\" (UniqueName: \"kubernetes.io/projected/cc776abf-676f-43e7-b4b5-7c68a29a7db7-kube-api-access-xrtf5\") pod \"node-ca-bmlcr\" (UID: \"cc776abf-676f-43e7-b4b5-7c68a29a7db7\") " pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.099594 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.099568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksgj\" (UniqueName: \"kubernetes.io/projected/25c6dddd-e2d2-43ab-8fc3-024497d8ac6d-kube-api-access-kksgj\") pod \"multus-additional-cni-plugins-prprc\" (UID: \"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d\") " pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.099695 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.099660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5bh\" (UniqueName: \"kubernetes.io/projected/6676a50f-361b-49f4-b68b-33549816ce4a-kube-api-access-2j5bh\") pod \"node-resolver-zrpwk\" (UID: \"6676a50f-361b-49f4-b68b-33549816ce4a\") " pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.099950 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.099925 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57tcl\" (UniqueName: \"kubernetes.io/projected/3d9ade42-fe48-40ae-b8ec-7dc429172620-kube-api-access-57tcl\") pod \"multus-kwxld\" (UID: \"3d9ade42-fe48-40ae-b8ec-7dc429172620\") " pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.100041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.100006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpp9v\" (UniqueName: \"kubernetes.io/projected/1a2a1f48-98c4-4f80-9e5b-4c66a2052840-kube-api-access-zpp9v\") pod \"aws-ebs-csi-driver-node-vkqm5\" (UID: \"1a2a1f48-98c4-4f80-9e5b-4c66a2052840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.100271 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.100249 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:50.100348 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.100276 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:50.100348 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.100291 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:50.100475 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.100366 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:50.6003477 +0000 UTC m=+2.060149981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:50.101731 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.101705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfvh\" (UniqueName: \"kubernetes.io/projected/4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64-kube-api-access-mgfvh\") pod \"ovnkube-node-8hsb7\" (UID: \"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64\") " pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.102219 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.102200 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwt8\" (UniqueName: \"kubernetes.io/projected/bad3f059-4b5e-4e0a-86ff-085088e730c0-kube-api-access-xwwt8\") pod \"iptables-alerter-npp7d\" (UID: \"bad3f059-4b5e-4e0a-86ff-085088e730c0\") " pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.102456 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.102438 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qzp\" (UniqueName: \"kubernetes.io/projected/691952e9-2ab0-4fce-be21-cf7bfe4be086-kube-api-access-85qzp\") pod \"tuned-s5gqv\" (UID: \"691952e9-2ab0-4fce-be21-cf7bfe4be086\") " pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.102531 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.102495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmnm\" (UniqueName: \"kubernetes.io/projected/e965746d-d914-436f-b115-0d3ebcc71e6d-kube-api-access-ppmnm\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:50.207158 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.207074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" Aug 05 16:22:50.217494 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.217467 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691952e9_2ab0_4fce_be21_cf7bfe4be086.slice/crio-c79f6ee7880453cb185a64d8b0a82464ac12da16f2d1b36a81cdeebb43f9027e WatchSource:0}: Error finding container c79f6ee7880453cb185a64d8b0a82464ac12da16f2d1b36a81cdeebb43f9027e: Status 404 returned error can't find the container with id c79f6ee7880453cb185a64d8b0a82464ac12da16f2d1b36a81cdeebb43f9027e Aug 05 16:22:50.217494 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.217489 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrpwk" Aug 05 16:22:50.225667 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.225646 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6676a50f_361b_49f4_b68b_33549816ce4a.slice/crio-35276cd0e20f9174af773025dfdf77397ab055fcfc9bde2ddec9ceaea3941da0 WatchSource:0}: Error finding container 35276cd0e20f9174af773025dfdf77397ab055fcfc9bde2ddec9ceaea3941da0: Status 404 returned error can't find the container with id 35276cd0e20f9174af773025dfdf77397ab055fcfc9bde2ddec9ceaea3941da0 Aug 05 16:22:50.229710 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.229693 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmlcr" Aug 05 16:22:50.235871 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.235850 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-prprc" Aug 05 16:22:50.238272 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.238253 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc776abf_676f_43e7_b4b5_7c68a29a7db7.slice/crio-72d04d9c9b70fa3d5445b50408fb950357baf626433f5a886fcde45271418558 WatchSource:0}: Error finding container 72d04d9c9b70fa3d5445b50408fb950357baf626433f5a886fcde45271418558: Status 404 returned error can't find the container with id 72d04d9c9b70fa3d5445b50408fb950357baf626433f5a886fcde45271418558 Aug 05 16:22:50.239752 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.239733 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:22:50.244775 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.244754 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c6dddd_e2d2_43ab_8fc3_024497d8ac6d.slice/crio-9ecbc7487250c6c253dcd94a06dbea62237f3b93207debfc5289cc42d5b718b4 WatchSource:0}: Error finding container 9ecbc7487250c6c253dcd94a06dbea62237f3b93207debfc5289cc42d5b718b4: Status 404 returned error can't find the container with id 9ecbc7487250c6c253dcd94a06dbea62237f3b93207debfc5289cc42d5b718b4 Aug 05 16:22:50.249083 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.249061 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a8caac_3a10_4067_acee_50b4d7bccf8e.slice/crio-83fcd50a1e78f8054410c80da61b8a1b7c8d9a7b1270dd1415ba673c9d7ccbe5 WatchSource:0}: Error finding container 83fcd50a1e78f8054410c80da61b8a1b7c8d9a7b1270dd1415ba673c9d7ccbe5: Status 404 returned error can't find the container with id 83fcd50a1e78f8054410c80da61b8a1b7c8d9a7b1270dd1415ba673c9d7ccbe5 Aug 05 16:22:50.255130 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.255111 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" Aug 05 16:22:50.265566 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.265541 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2a1f48_98c4_4f80_9e5b_4c66a2052840.slice/crio-b5f3da3b57ab4ab06a6606470ec831ab8d7d5943ea33b4d6389816748b0bb713 WatchSource:0}: Error finding container b5f3da3b57ab4ab06a6606470ec831ab8d7d5943ea33b4d6389816748b0bb713: Status 404 returned error can't find the container with id b5f3da3b57ab4ab06a6606470ec831ab8d7d5943ea33b4d6389816748b0bb713 Aug 05 16:22:50.284901 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.284881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-npp7d" Aug 05 16:22:50.292619 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.292595 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad3f059_4b5e_4e0a_86ff_085088e730c0.slice/crio-cd21690c5a5c52084e099aaff108b9ba6c1b5cd96fef6c0fb901511805ece47a WatchSource:0}: Error finding container cd21690c5a5c52084e099aaff108b9ba6c1b5cd96fef6c0fb901511805ece47a: Status 404 returned error can't find the container with id cd21690c5a5c52084e099aaff108b9ba6c1b5cd96fef6c0fb901511805ece47a Aug 05 16:22:50.298272 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.298257 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:22:50.305783 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.305751 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwxld" Aug 05 16:22:50.305995 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.305974 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1229c1_ae68_4c83_8cde_7b1c7b2a4c64.slice/crio-79cb6ed9ae7ace4a69aff037be13bf068237d9437f6cbd921ac09559ca2460c9 WatchSource:0}: Error finding container 79cb6ed9ae7ace4a69aff037be13bf068237d9437f6cbd921ac09559ca2460c9: Status 404 returned error can't find the container with id 79cb6ed9ae7ace4a69aff037be13bf068237d9437f6cbd921ac09559ca2460c9 Aug 05 16:22:50.317911 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:22:50.317891 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9ade42_fe48_40ae_b8ec_7dc429172620.slice/crio-65e6d1977d3bcdca001ad881aac8f48abd50ec0ef1de56d21a4968209aecc5a1 WatchSource:0}: Error finding container 65e6d1977d3bcdca001ad881aac8f48abd50ec0ef1de56d21a4968209aecc5a1: Status 404 returned error can't find the container with id 65e6d1977d3bcdca001ad881aac8f48abd50ec0ef1de56d21a4968209aecc5a1 Aug 05 16:22:50.463539 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.463473 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:50.591424 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.591374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:50.591613 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.591433 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:50.591613 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.591583 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:50.591613 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.591585 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:50.591816 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.591652 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:22:51.591631618 +0000 UTC m=+3.051433897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:50.591816 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.591673 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:51.59166292 +0000 UTC m=+3.051465182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:50.692176 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.692134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:50.692371 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.692346 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:50.692450 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.692379 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:50.692450 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.692393 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:50.692569 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:50.692452 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:51.692431465 +0000 UTC m=+3.152233744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:50.793134 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.793057 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:50.919913 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.919849 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 16:17:50 +0000 UTC" deadline="2028-04-20 22:36:59.806601836 +0000 UTC" Aug 05 16:22:50.919913 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:50.919911 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14982h14m8.886694975s" Aug 05 16:22:51.005734 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.005574 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:51.005734 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.005701 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:22:51.023493 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.023435 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrpwk" event={"ID":"6676a50f-361b-49f4-b68b-33549816ce4a","Type":"ContainerStarted","Data":"35276cd0e20f9174af773025dfdf77397ab055fcfc9bde2ddec9ceaea3941da0"} Aug 05 16:22:51.035490 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.035450 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-npp7d" event={"ID":"bad3f059-4b5e-4e0a-86ff-085088e730c0","Type":"ContainerStarted","Data":"cd21690c5a5c52084e099aaff108b9ba6c1b5cd96fef6c0fb901511805ece47a"} Aug 05 16:22:51.045271 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.045159 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5ckqd" event={"ID":"18a8caac-3a10-4067-acee-50b4d7bccf8e","Type":"ContainerStarted","Data":"83fcd50a1e78f8054410c80da61b8a1b7c8d9a7b1270dd1415ba673c9d7ccbe5"} Aug 05 16:22:51.054402 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.054369 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmlcr" event={"ID":"cc776abf-676f-43e7-b4b5-7c68a29a7db7","Type":"ContainerStarted","Data":"72d04d9c9b70fa3d5445b50408fb950357baf626433f5a886fcde45271418558"} Aug 05 16:22:51.070566 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.070534 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" event={"ID":"691952e9-2ab0-4fce-be21-cf7bfe4be086","Type":"ContainerStarted","Data":"c79f6ee7880453cb185a64d8b0a82464ac12da16f2d1b36a81cdeebb43f9027e"} Aug 05 16:22:51.081942 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.081909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" event={"ID":"39a7c3613bf8e5e7fd5e0d2f670dcc7b","Type":"ContainerStarted","Data":"d4548cd1ed38dbc14a6d7137bdc97861d75aa78b18a538c91a6842a6398d126e"} Aug 05 16:22:51.086051 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.086021 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" event={"ID":"6718bd51b27ca92571804b6ccc53294d","Type":"ContainerStarted","Data":"ea9f47ce3b0e3b9ed3ca13a056766519579ca711ea5dcaf7dc45caf7d17a2692"} Aug 05 16:22:51.100638 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.100600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwxld" event={"ID":"3d9ade42-fe48-40ae-b8ec-7dc429172620","Type":"ContainerStarted","Data":"65e6d1977d3bcdca001ad881aac8f48abd50ec0ef1de56d21a4968209aecc5a1"} Aug 05 16:22:51.109936 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.109906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"79cb6ed9ae7ace4a69aff037be13bf068237d9437f6cbd921ac09559ca2460c9"} Aug 05 16:22:51.115695 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.115439 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" event={"ID":"1a2a1f48-98c4-4f80-9e5b-4c66a2052840","Type":"ContainerStarted","Data":"b5f3da3b57ab4ab06a6606470ec831ab8d7d5943ea33b4d6389816748b0bb713"} Aug 05 16:22:51.129060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.129028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerStarted","Data":"9ecbc7487250c6c253dcd94a06dbea62237f3b93207debfc5289cc42d5b718b4"} Aug 05 16:22:51.598509 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.598469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:51.598689 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.598529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:51.598689 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.598678 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:51.598823 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.598745 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:22:53.598726982 +0000 UTC m=+5.058529264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:51.599343 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.599176 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:51.599343 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.599235 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:53.599218463 +0000 UTC m=+5.059020731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:51.700070 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.700034 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:51.700284 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.700247 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:51.700284 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.700272 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:51.700284 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.700285 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:51.700455 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:51.700354 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:53.700334535 +0000 UTC m=+5.160136814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:51.857722 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.857639 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:22:51.921118 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.921031 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 16:17:50 +0000 UTC" deadline="2028-01-07 01:52:58.48224379 +0000 UTC" Aug 05 16:22:51.921118 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:51.921071 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12465h30m6.561178028s" Aug 05 16:22:52.006323 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:52.006263 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:52.006598 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:52.006417 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:22:52.006857 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:52.006836 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:52.006968 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:52.006953 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:22:53.008762 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:53.005792 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:53.008762 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.005926 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:22:53.619387 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:53.619248 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:53.619387 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:53.619314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:53.619812 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.619455 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:53.619812 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.619526 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:22:57.619503045 +0000 UTC m=+9.079305330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:53.620015 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.619991 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:53.620084 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.620058 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:57.62004177 +0000 UTC m=+9.079844046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:53.720465 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:53.720311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:53.720656 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.720521 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:53.720656 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.720549 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:53.720656 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.720564 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:53.720656 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:53.720630 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:57.720608008 +0000 UTC m=+9.180410270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:54.006058 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:54.005973 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:54.006236 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:54.006149 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:22:54.006794 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:54.006577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:54.006794 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:54.006697 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:22:55.006399 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:55.006358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:55.006877 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:55.006498 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:22:56.005845 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:56.005804 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:56.006031 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:56.005806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:56.006031 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:56.005931 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:22:56.006031 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:56.006010 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:22:57.006001 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:57.005964 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:57.006449 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.006114 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:22:57.651609 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:57.651562 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:57.651783 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:57.651628 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:57.651783 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.651755 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:57.651918 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.651815 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:23:05.651800079 +0000 UTC m=+17.111602343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:22:57.652249 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.652225 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:57.652343 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.652292 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:05.652279486 +0000 UTC m=+17.112081748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:22:57.752360 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:57.752311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:57.752527 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.752489 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:22:57.752527 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.752508 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:22:57.752527 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.752520 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:57.752699 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:57.752577 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:05.752561517 +0000 UTC m=+17.212363776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:22:58.006215 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:58.006138 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:22:58.006652 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:58.006272 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:22:58.006652 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:58.006333 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:22:58.006652 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:58.006461 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:22:59.006938 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:22:59.006907 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:22:59.007402 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:22:59.007016 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:00.006255 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:00.006224 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:00.006255 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:00.006261 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:00.006507 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:00.006355 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:00.006577 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:00.006550 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:01.005912 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:01.005874 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:01.006383 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:01.006039 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:02.005956 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:02.005927 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:02.005956 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:02.005949 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:02.006413 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:02.006058 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:02.006413 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:02.006174 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:03.005617 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:03.005580 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:03.005781 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:03.005716 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:04.005585 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:04.005550 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:04.006018 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:04.005693 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:04.006018 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:04.005753 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:04.006018 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:04.005866 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:05.006253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:05.006219 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:05.006640 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.006341 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:05.712947 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:05.712920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:05.713155 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:05.712953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:05.713155 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.713057 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:05.713155 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.713057 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:05.713155 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.713131 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:23:21.713117649 +0000 UTC m=+33.172919907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:05.713155 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.713146 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:21.713140061 +0000 UTC m=+33.172942319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:05.813540 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:05.813515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:05.813683 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.813665 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:05.813683 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.813684 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:05.813758 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.813694 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:05.813758 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:05.813742 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:21.813726129 +0000 UTC m=+33.273528406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:06.005526 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:06.005455 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:06.005714 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:06.005568 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:06.005714 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:06.005622 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:06.005817 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:06.005733 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:07.008839 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:07.008808 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:07.009295 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:07.008926 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:08.005696 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:08.005655 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:08.005880 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:08.005669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:08.005880 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:08.005808 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:08.005880 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:08.005838 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:09.007156 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:09.007120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:09.007593 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:09.007237 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:10.005534 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.005510 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:10.005618 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.005515 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:10.005655 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:10.005614 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:10.005702 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:10.005684 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:10.190386 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.190220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwxld" event={"ID":"3d9ade42-fe48-40ae-b8ec-7dc429172620","Type":"ContainerStarted","Data":"2829e34b7449138f1c8b2150b1a78b7bb0bfcca4588306d41e117b16abbd3927"} Aug 05 16:23:10.192569 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.192548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"2b428e287ed6007df9579a88de0b2d487c56201107fbc26edbb7ff0145217253"} Aug 05 16:23:10.192638 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.192610 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"a3d34d2067d94d4183e1bb9a04492281cf35d0821c941645bd61bd8b9292edb0"} Aug 05 16:23:10.192695 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.192643 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"69a67933f9c75c2517398fc87c80453031df2354a6c576833964cd5b2b4c264c"} Aug 05 16:23:10.192695 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.192673 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"7aef9a792967f018f0d28d4a1a9581f5575dcf7ec01d51cb0ffc0a89df8ece4c"} Aug 05 16:23:10.193804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.193774 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" event={"ID":"691952e9-2ab0-4fce-be21-cf7bfe4be086","Type":"ContainerStarted","Data":"9138fef2e1cc1dd2fcf410aedf5d08734809590cf25304062f30939c6f14c0bc"} Aug 05 16:23:10.194835 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.194817 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" event={"ID":"39a7c3613bf8e5e7fd5e0d2f670dcc7b","Type":"ContainerStarted","Data":"97b319b82b4576b6f783648d96d3900bc82216ff3231e2b98b5886c5869edb3b"} Aug 05 16:23:10.214398 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.214359 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kwxld" podStartSLOduration=1.8441504960000001 podStartE2EDuration="21.214345887s" totalImagesPullingTime="19.370195391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:10.213229989 +0000 UTC m=+21.673032295" watchObservedRunningTime="2026-08-05 16:23:10.214345887 +0000 UTC m=+21.674148167" Aug 05 16:23:10.226461 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.226419 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-60.ec2.internal" podStartSLOduration=20.226406073 podStartE2EDuration="20.226406073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:10.226387746 +0000 UTC m=+21.686190026" watchObservedRunningTime="2026-08-05 16:23:10.226406073 +0000 UTC m=+21.686208351" Aug 05 16:23:10.242471 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:10.242432 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s5gqv" podStartSLOduration=1.810526182 podStartE2EDuration="21.242419866s" totalImagesPullingTime="19.431893684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:10.24227712 +0000 UTC m=+21.702079401" watchObservedRunningTime="2026-08-05 16:23:10.242419866 +0000 UTC m=+21.702222138" Aug 05 16:23:11.005534 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.005510 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:11.005660 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:11.005636 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:11.199598 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.199396 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:23:11.200253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.199886 2567 generic.go:356] "Generic (PLEG): container finished" podID="4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64" containerID="69a67933f9c75c2517398fc87c80453031df2354a6c576833964cd5b2b4c264c" exitCode=1 Aug 05 16:23:11.200253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.199955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerDied","Data":"69a67933f9c75c2517398fc87c80453031df2354a6c576833964cd5b2b4c264c"} Aug 05 16:23:11.200253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.200035 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"fa2af841a6499068697f6a24228199180b43c918f261f258e2ca3b2e8b864787"} Aug 05 16:23:11.200253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.200077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"aec427b21e18ae32520c14ff3a0b941b5857bacd893e34e002afd73278f2c59d"} Aug 05 16:23:11.201291 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.201272 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" event={"ID":"1a2a1f48-98c4-4f80-9e5b-4c66a2052840","Type":"ContainerStarted","Data":"d96abb46f209ee78e1d31f533f328aec76f29364b8c624d42bb149ff83a0c317"} Aug 05 16:23:11.202580 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.202555 2567 generic.go:356] "Generic (PLEG): container finished" podID="25c6dddd-e2d2-43ab-8fc3-024497d8ac6d" containerID="94a3b042f9f3257c1758e5bbc43d4b4d286548074de86e1a9ed2fd8285dacc6a" exitCode=0 Aug 05 16:23:11.202671 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.202623 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerDied","Data":"94a3b042f9f3257c1758e5bbc43d4b4d286548074de86e1a9ed2fd8285dacc6a"} Aug 05 16:23:11.203990 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.203967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrpwk" event={"ID":"6676a50f-361b-49f4-b68b-33549816ce4a","Type":"ContainerStarted","Data":"b9ae7b5b9e3083e6b8cf3e6175b416a6eec1c7920f4e34a1fefed478b970548d"} Aug 05 16:23:11.205417 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.205394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-npp7d" event={"ID":"bad3f059-4b5e-4e0a-86ff-085088e730c0","Type":"ContainerStarted","Data":"4ae9c4efaa597759f76400175ea2d83d5e9a6d542a62f35ce7215bf77e80d962"} Aug 05 16:23:11.206703 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.206684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5ckqd" event={"ID":"18a8caac-3a10-4067-acee-50b4d7bccf8e","Type":"ContainerStarted","Data":"6715dfb8efbba9a24ca8289b43cf80f13b9e9879aea8b7b48fa721ff1e1be016"} Aug 05 16:23:11.207955 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.207937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmlcr" event={"ID":"cc776abf-676f-43e7-b4b5-7c68a29a7db7","Type":"ContainerStarted","Data":"4fea02c3cee477dd3f905f39cb554ed80a265e861dd571ff5308eebc9921ca98"} Aug 05 16:23:11.209183 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.209164 2567 generic.go:356] "Generic (PLEG): container finished" podID="6718bd51b27ca92571804b6ccc53294d" containerID="7a414e7e42bcd3df5aafcde8704f1793483df4a6bb16276e3abea91e25817f9f" exitCode=0 Aug 05 16:23:11.209266 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.209222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" event={"ID":"6718bd51b27ca92571804b6ccc53294d","Type":"ContainerDied","Data":"7a414e7e42bcd3df5aafcde8704f1793483df4a6bb16276e3abea91e25817f9f"} Aug 05 16:23:11.248942 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.248909 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5ckqd" podStartSLOduration=2.85754938 podStartE2EDuration="22.248895385s" totalImagesPullingTime="19.391346005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:11.248440115 +0000 UTC m=+22.708242396" watchObservedRunningTime="2026-08-05 16:23:11.248895385 +0000 UTC m=+22.708697667" Aug 05 16:23:11.267142 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.267060 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zrpwk" podStartSLOduration=2.8521809190000003 podStartE2EDuration="22.267047506s" totalImagesPullingTime="19.414866587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:11.266942978 +0000 UTC m=+22.726745260" watchObservedRunningTime="2026-08-05 16:23:11.267047506 +0000 UTC m=+22.726849786" Aug 05 16:23:11.281967 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.281934 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bmlcr" podStartSLOduration=2.879739687 podStartE2EDuration="22.281921949s" totalImagesPullingTime="19.402182262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:11.281636436 +0000 UTC m=+22.741438716" watchObservedRunningTime="2026-08-05 16:23:11.281921949 +0000 UTC m=+22.741724229" Aug 05 16:23:11.295764 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.295732 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-npp7d" podStartSLOduration=2.938621007 podStartE2EDuration="22.295723158s" totalImagesPullingTime="19.357102151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:11.295468826 +0000 UTC m=+22.755271106" watchObservedRunningTime="2026-08-05 16:23:11.295723158 +0000 UTC m=+22.755525439" Aug 05 16:23:11.441698 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.441663 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5ckqd_18a8caac-3a10-4067-acee-50b4d7bccf8e/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:11.506257 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:11.506188 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5ckqd_18a8caac-3a10-4067-acee-50b4d7bccf8e/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:12.006125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.006082 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:12.006330 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:12.006208 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:12.006330 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.006238 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:12.006330 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:12.006314 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:12.214311 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.214275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" event={"ID":"6718bd51b27ca92571804b6ccc53294d","Type":"ContainerStarted","Data":"5bc221de9c3d43b6796e669997b23a80b829cdfef7fa7eadd20e0e7ed357bdb9"} Aug 05 16:23:12.228228 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.228182 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-60.ec2.internal" podStartSLOduration=22.228165976 podStartE2EDuration="22.228165976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:12.227813754 +0000 UTC m=+23.687616032" watchObservedRunningTime="2026-08-05 16:23:12.228165976 +0000 UTC m=+23.687968259" Aug 05 16:23:12.300741 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.300717 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 05 16:23:12.944710 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.944595 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T16:23:12.300737311Z","UUID":"735ff248-2083-47bd-81ae-78998f4c3f07","Handler":null,"Name":"","Endpoint":""} Aug 05 16:23:12.946020 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.945985 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 05 16:23:12.946163 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.946026 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 05 16:23:12.997060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:12.997033 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:23:13.006481 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:13.006455 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:13.006620 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:13.006580 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:13.220507 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:13.220447 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:23:13.220927 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:13.220893 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"02e0f11b2ca0a86915b5307f69d9fcc2493cb371e0bff393de1576d272e82ef1"} Aug 05 16:23:13.222869 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:13.222833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" event={"ID":"1a2a1f48-98c4-4f80-9e5b-4c66a2052840","Type":"ContainerStarted","Data":"dd894dfd5bdd8aa588d90e5e3aec2fff019f8bfe2481f8f51f26c54a384a436d"} Aug 05 16:23:14.005970 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:14.005804 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:14.006252 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:14.005809 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:14.006252 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:14.006061 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:14.006252 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:14.006180 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:14.226888 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:14.226850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" event={"ID":"1a2a1f48-98c4-4f80-9e5b-4c66a2052840","Type":"ContainerStarted","Data":"d2cef5aa488e63ae4e568507dd80bff3b5755811626929a2c5e2490fe76dee70"} Aug 05 16:23:14.242713 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:14.242672 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vkqm5" podStartSLOduration=3.8039047 podStartE2EDuration="25.242658036s" totalImagesPullingTime="21.438753336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:14.24257371 +0000 UTC m=+25.702375992" watchObservedRunningTime="2026-08-05 16:23:14.242658036 +0000 UTC m=+25.702460309" Aug 05 16:23:14.600003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:14.599960 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:23:14.600709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:14.600690 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:23:15.006378 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:15.006299 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:15.006521 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:15.006422 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:15.229611 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:15.229586 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5ckqd" Aug 05 16:23:15.290429 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:15.290351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5ckqd_18a8caac-3a10-4067-acee-50b4d7bccf8e/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:16.005623 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.005477 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:16.005729 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.005490 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:16.005770 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:16.005730 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:16.005770 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:16.005761 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:16.233024 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.232938 2567 generic.go:356] "Generic (PLEG): container finished" podID="25c6dddd-e2d2-43ab-8fc3-024497d8ac6d" containerID="3e530f52b0fb58dde9d219926731e126b6de178db1ee4fb3dcb93b2e5412d6df" exitCode=0 Aug 05 16:23:16.233742 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.233017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerDied","Data":"3e530f52b0fb58dde9d219926731e126b6de178db1ee4fb3dcb93b2e5412d6df"} Aug 05 16:23:16.236298 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.236282 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:23:16.236679 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.236631 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"ecbc5ce8bfff97e13158ae7e3b893969fdb9769bbbb42c24ec370df27366e06e"} Aug 05 16:23:16.236951 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.236934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:23:16.237039 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.237001 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:23:16.237114 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.237041 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:23:16.237114 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.237053 2567 scope.go:117] "RemoveContainer" containerID="69a67933f9c75c2517398fc87c80453031df2354a6c576833964cd5b2b4c264c" Aug 05 16:23:16.252569 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.252551 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:23:16.252662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:16.252638 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:23:17.010870 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.008284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:17.010870 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:17.008409 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:17.242636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.242567 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:23:17.243101 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.242910 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" event={"ID":"4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64","Type":"ContainerStarted","Data":"72ee78041f1eb1cbf4d3b2b3196fe85e5a7634bc06407e2d014ed50fefa6be0c"} Aug 05 16:23:17.273434 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.273384 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" podStartSLOduration=8.84507162 podStartE2EDuration="28.273356647s" totalImagesPullingTime="19.428285027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:17.272269781 +0000 UTC m=+28.732072075" watchObservedRunningTime="2026-08-05 16:23:17.273356647 +0000 UTC m=+28.733158927" Aug 05 16:23:17.920699 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.920666 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qs6sj"] Aug 05 16:23:17.920888 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.920817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:17.920952 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:17.920918 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:17.921258 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.921232 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kjwrw"] Aug 05 16:23:17.921396 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.921328 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:17.921447 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:17.921421 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:17.931521 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.931495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ctr2g"] Aug 05 16:23:17.931609 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:17.931575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:17.931686 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:17.931664 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:18.247128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:18.247040 2567 generic.go:356] "Generic (PLEG): container finished" podID="25c6dddd-e2d2-43ab-8fc3-024497d8ac6d" containerID="b1c2077533d0d07a0950c88390db5325e7a8ae470e3c22e10698cae4da62ce9e" exitCode=0 Aug 05 16:23:18.247468 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:18.247130 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerDied","Data":"b1c2077533d0d07a0950c88390db5325e7a8ae470e3c22e10698cae4da62ce9e"} Aug 05 16:23:20.006324 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:20.006112 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:20.006733 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:20.006118 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:20.006733 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:20.006381 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:20.006733 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:20.006151 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:20.006733 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:20.006466 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:20.006733 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:20.006524 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:20.253150 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:20.253120 2567 generic.go:356] "Generic (PLEG): container finished" podID="25c6dddd-e2d2-43ab-8fc3-024497d8ac6d" containerID="4368141c3463fc0deb0ef72ed327c2783234be03ead6458c7ee1ac2a0438c249" exitCode=0 Aug 05 16:23:20.253297 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:20.253180 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerDied","Data":"4368141c3463fc0deb0ef72ed327c2783234be03ead6458c7ee1ac2a0438c249"} Aug 05 16:23:21.735799 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:21.735766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:21.736392 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:21.735810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:21.736392 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.735933 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:21.736392 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.735954 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:21.736392 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.735993 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs podName:e965746d-d914-436f-b115-0d3ebcc71e6d nodeName:}" failed. No retries permitted until 2026-08-05 16:23:53.735977998 +0000 UTC m=+65.195780256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs") pod "network-metrics-daemon-qs6sj" (UID: "e965746d-d914-436f-b115-0d3ebcc71e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:23:21.736392 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.736011 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret podName:5e464b10-9c26-458f-a3e6-5effcba181f5 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:53.736002113 +0000 UTC m=+65.195804373 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret") pod "global-pull-secret-syncer-kjwrw" (UID: "5e464b10-9c26-458f-a3e6-5effcba181f5") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:23:21.836770 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:21.836741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:21.836952 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.836897 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:23:21.836952 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.836921 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:23:21.836952 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.836930 2567 projected.go:196] Error preparing data for projected volume kube-api-access-f9mvj for pod openshift-network-diagnostics/network-check-target-ctr2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:21.837130 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:21.836981 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj podName:68255723-2c55-4175-a70b-b337c49e9f09 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:53.83696567 +0000 UTC m=+65.296767928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f9mvj" (UniqueName: "kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj") pod "network-check-target-ctr2g" (UID: "68255723-2c55-4175-a70b-b337c49e9f09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:23:22.008254 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.005611 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:22.008254 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.005709 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:22.008254 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:22.005849 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kjwrw" podUID="5e464b10-9c26-458f-a3e6-5effcba181f5" Aug 05 16:23:22.008254 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:22.006412 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs6sj" podUID="e965746d-d914-436f-b115-0d3ebcc71e6d" Aug 05 16:23:22.008254 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.006499 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:22.008254 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:22.006595 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ctr2g" podUID="68255723-2c55-4175-a70b-b337c49e9f09" Aug 05 16:23:22.872283 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.872253 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-60.ec2.internal" event="NodeReady" Aug 05 16:23:22.872729 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.872394 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 05 16:23:22.929669 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.929638 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dht42"] Aug 05 16:23:22.961181 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.961149 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vv5rs"] Aug 05 16:23:22.961351 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.961297 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dht42" Aug 05 16:23:22.963167 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.963143 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 05 16:23:22.963274 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.963224 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 05 16:23:22.963682 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.963666 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w85qf\"" Aug 05 16:23:22.975136 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.975115 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8566dd4b4b-fzhgr"] Aug 05 16:23:22.975245 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.975158 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:22.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.977247 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-chrqx\"" Aug 05 16:23:22.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.977333 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 05 16:23:22.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.977375 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 05 16:23:22.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.977484 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 05 16:23:22.995223 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.995199 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vv5rs"] Aug 05 16:23:22.995315 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.995228 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dht42"] Aug 05 16:23:22.995315 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.995244 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8566dd4b4b-fzhgr"] Aug 05 16:23:22.995390 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.995322 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:22.997123 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.997106 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 05 16:23:22.997484 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.997468 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kcq9m\"" Aug 05 16:23:22.997586 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.997549 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 05 16:23:22.997661 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:22.997643 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 05 16:23:23.001873 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.001852 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 05 16:23:23.046830 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.046804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bl46\" (UniqueName: \"kubernetes.io/projected/f0757141-204c-4f0d-b1af-5c515b125c42-kube-api-access-6bl46\") pod \"ingress-canary-vv5rs\" (UID: \"f0757141-204c-4f0d-b1af-5c515b125c42\") " pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.046978 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.046839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.046978 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.046866 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0757141-204c-4f0d-b1af-5c515b125c42-cert\") pod \"ingress-canary-vv5rs\" (UID: \"f0757141-204c-4f0d-b1af-5c515b125c42\") " pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.047117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.047000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62cc78cf-1569-4119-9ad6-5fe13239a970-config-volume\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.047117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.047035 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62cc78cf-1569-4119-9ad6-5fe13239a970-tmp-dir\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.047117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.047104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-922sn\" (UniqueName: \"kubernetes.io/projected/62cc78cf-1569-4119-9ad6-5fe13239a970-kube-api-access-922sn\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.147802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.147731 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0757141-204c-4f0d-b1af-5c515b125c42-cert\") pod \"ingress-canary-vv5rs\" (UID: \"f0757141-204c-4f0d-b1af-5c515b125c42\") " pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.147802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.147764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-bound-sa-token\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.147993 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.147933 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62cc78cf-1569-4119-9ad6-5fe13239a970-config-volume\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.147993 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.147963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.147993 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.147986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-ca-trust-extracted\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.148150 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148014 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62cc78cf-1569-4119-9ad6-5fe13239a970-tmp-dir\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.148150 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-922sn\" (UniqueName: \"kubernetes.io/projected/62cc78cf-1569-4119-9ad6-5fe13239a970-kube-api-access-922sn\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.148150 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-trusted-ca\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.148150 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-image-registry-private-configuration\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.148320 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpns\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-kube-api-access-rnpns\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.148320 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-certificates\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.148320 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-installation-pull-secrets\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.148320 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bl46\" (UniqueName: \"kubernetes.io/projected/f0757141-204c-4f0d-b1af-5c515b125c42-kube-api-access-6bl46\") pod \"ingress-canary-vv5rs\" (UID: \"f0757141-204c-4f0d-b1af-5c515b125c42\") " pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.148522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148338 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.148522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148348 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62cc78cf-1569-4119-9ad6-5fe13239a970-tmp-dir\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.148522 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.148450 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:23.148522 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.148520 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls podName:62cc78cf-1569-4119-9ad6-5fe13239a970 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:23.648503218 +0000 UTC m=+35.108305492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls") pod "dns-default-dht42" (UID: "62cc78cf-1569-4119-9ad6-5fe13239a970") : secret "dns-default-metrics-tls" not found Aug 05 16:23:23.148730 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.148609 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62cc78cf-1569-4119-9ad6-5fe13239a970-config-volume\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.152798 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.152774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0757141-204c-4f0d-b1af-5c515b125c42-cert\") pod \"ingress-canary-vv5rs\" (UID: \"f0757141-204c-4f0d-b1af-5c515b125c42\") " pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.156434 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.156416 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-922sn\" (UniqueName: \"kubernetes.io/projected/62cc78cf-1569-4119-9ad6-5fe13239a970-kube-api-access-922sn\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.156954 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.156923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bl46\" (UniqueName: \"kubernetes.io/projected/f0757141-204c-4f0d-b1af-5c515b125c42-kube-api-access-6bl46\") pod \"ingress-canary-vv5rs\" (UID: \"f0757141-204c-4f0d-b1af-5c515b125c42\") " pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.249294 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-trusted-ca\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249451 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-image-registry-private-configuration\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249451 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpns\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-kube-api-access-rnpns\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249451 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-certificates\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249451 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249398 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-installation-pull-secrets\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249644 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-bound-sa-token\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249644 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.249644 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-ca-trust-extracted\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.250126 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.249919 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-ca-trust-extracted\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.250126 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.249939 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 16:23:23.250126 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.249959 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8566dd4b4b-fzhgr: secret "image-registry-tls" not found Aug 05 16:23:23.250126 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.250030 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls podName:b17f2c80-c696-4cc8-bd0a-a231d3c1f203 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:23.750010515 +0000 UTC m=+35.209812822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls") pod "image-registry-8566dd4b4b-fzhgr" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203") : secret "image-registry-tls" not found Aug 05 16:23:23.250385 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.250251 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-trusted-ca\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.250434 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.250384 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-certificates\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.252456 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.252428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-installation-pull-secrets\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.252735 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.252711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-image-registry-private-configuration\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.261479 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.261455 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpns\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-kube-api-access-rnpns\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.261693 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.261669 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-bound-sa-token\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.285860 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.285842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vv5rs" Aug 05 16:23:23.474241 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.474046 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vv5rs"] Aug 05 16:23:23.653145 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.653026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:23.653318 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.653210 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:23.653318 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.653305 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls podName:62cc78cf-1569-4119-9ad6-5fe13239a970 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:24.653285348 +0000 UTC m=+36.113087623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls") pod "dns-default-dht42" (UID: "62cc78cf-1569-4119-9ad6-5fe13239a970") : secret "dns-default-metrics-tls" not found Aug 05 16:23:23.754408 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:23.754374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:23.754570 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.754549 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 16:23:23.754570 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.754568 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8566dd4b4b-fzhgr: secret "image-registry-tls" not found Aug 05 16:23:23.754661 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:23.754631 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls podName:b17f2c80-c696-4cc8-bd0a-a231d3c1f203 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:24.754611793 +0000 UTC m=+36.214414054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls") pod "image-registry-8566dd4b4b-fzhgr" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203") : secret "image-registry-tls" not found Aug 05 16:23:24.005997 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.005913 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:24.005997 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.005937 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:24.006696 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.006043 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:24.009044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.008573 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 16:23:24.009044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.008657 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 16:23:24.009044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.008694 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t7dww\"" Aug 05 16:23:24.009044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.008769 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 16:23:24.009044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.008929 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 16:23:24.009044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.009019 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zm8j\"" Aug 05 16:23:24.661754 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.661713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:24.661932 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:24.661857 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:24.661932 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:24.661921 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls podName:62cc78cf-1569-4119-9ad6-5fe13239a970 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:26.661903563 +0000 UTC m=+38.121705843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls") pod "dns-default-dht42" (UID: "62cc78cf-1569-4119-9ad6-5fe13239a970") : secret "dns-default-metrics-tls" not found Aug 05 16:23:24.762144 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:24.762106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:24.762359 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:24.762255 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 16:23:24.762359 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:24.762271 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8566dd4b4b-fzhgr: secret "image-registry-tls" not found Aug 05 16:23:24.762359 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:24.762339 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls podName:b17f2c80-c696-4cc8-bd0a-a231d3c1f203 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:26.76231812 +0000 UTC m=+38.222120411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls") pod "image-registry-8566dd4b4b-fzhgr" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203") : secret "image-registry-tls" not found Aug 05 16:23:24.993181 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:24.993113 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0757141_204c_4f0d_b1af_5c515b125c42.slice/crio-21d3ecc3eae21566de47b02c655ea64bace1081d347c8bcaf563491d49029d9b WatchSource:0}: Error finding container 21d3ecc3eae21566de47b02c655ea64bace1081d347c8bcaf563491d49029d9b: Status 404 returned error can't find the container with id 21d3ecc3eae21566de47b02c655ea64bace1081d347c8bcaf563491d49029d9b Aug 05 16:23:25.265708 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:25.265607 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vv5rs" event={"ID":"f0757141-204c-4f0d-b1af-5c515b125c42","Type":"ContainerStarted","Data":"21d3ecc3eae21566de47b02c655ea64bace1081d347c8bcaf563491d49029d9b"} Aug 05 16:23:26.270537 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:26.270507 2567 generic.go:356] "Generic (PLEG): container finished" podID="25c6dddd-e2d2-43ab-8fc3-024497d8ac6d" containerID="c94e8398bbcd63bac1f7ea5ce73a7314e09b77cb606644c89a30b68bde0f8bb1" exitCode=0 Aug 05 16:23:26.271038 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:26.270545 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerDied","Data":"c94e8398bbcd63bac1f7ea5ce73a7314e09b77cb606644c89a30b68bde0f8bb1"} Aug 05 16:23:26.675801 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:26.675764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:26.675976 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:26.675940 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:26.676040 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:26.676025 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls podName:62cc78cf-1569-4119-9ad6-5fe13239a970 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.67600292 +0000 UTC m=+42.135805198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls") pod "dns-default-dht42" (UID: "62cc78cf-1569-4119-9ad6-5fe13239a970") : secret "dns-default-metrics-tls" not found Aug 05 16:23:26.776473 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:26.776444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:26.776631 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:26.776609 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 16:23:26.776631 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:26.776626 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8566dd4b4b-fzhgr: secret "image-registry-tls" not found Aug 05 16:23:26.776760 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:26.776685 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls podName:b17f2c80-c696-4cc8-bd0a-a231d3c1f203 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.776666217 +0000 UTC m=+42.236468493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls") pod "image-registry-8566dd4b4b-fzhgr" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203") : secret "image-registry-tls" not found Aug 05 16:23:27.276345 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:27.276308 2567 generic.go:356] "Generic (PLEG): container finished" podID="25c6dddd-e2d2-43ab-8fc3-024497d8ac6d" containerID="89da486f35a5113ba2afcdbca8c2a55c0002267525fd7ad3df990bd9fd5cf132" exitCode=0 Aug 05 16:23:27.276739 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:27.276359 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerDied","Data":"89da486f35a5113ba2afcdbca8c2a55c0002267525fd7ad3df990bd9fd5cf132"} Aug 05 16:23:28.282317 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:28.282105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prprc" event={"ID":"25c6dddd-e2d2-43ab-8fc3-024497d8ac6d","Type":"ContainerStarted","Data":"291fac3a1d2f8e8be9882239a4e7428613c409554a781671fec81b6670b5e3b8"} Aug 05 16:23:28.283573 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:28.283549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vv5rs" event={"ID":"f0757141-204c-4f0d-b1af-5c515b125c42","Type":"ContainerStarted","Data":"2434c9c44c07457739e8d066477c04adcc10dc20b8f7e51e234c6753d36d0c08"} Aug 05 16:23:28.303278 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:28.303236 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-prprc" podStartSLOduration=7.611150082 podStartE2EDuration="39.303220471s" totalImagesPullingTime="31.692070389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:28.30228574 +0000 UTC m=+39.762088021" watchObservedRunningTime="2026-08-05 16:23:28.303220471 +0000 UTC m=+39.763022751" Aug 05 16:23:28.317206 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:28.317164 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vv5rs" podStartSLOduration=3.626577105 podStartE2EDuration="6.317152296s" totalImagesPullingTime="2.690575191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:28.316833118 +0000 UTC m=+39.776635400" watchObservedRunningTime="2026-08-05 16:23:28.317152296 +0000 UTC m=+39.776954581" Aug 05 16:23:29.162620 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.160964 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9pmrz"] Aug 05 16:23:29.185123 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.185082 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9pmrz"] Aug 05 16:23:29.185264 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.185177 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.187114 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.187069 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 16:23:29.187114 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.187072 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 16:23:29.187295 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.187278 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 16:23:29.187350 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.187307 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 16:23:29.187388 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.187312 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-422vm\"" Aug 05 16:23:29.295540 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.295508 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.295540 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.295537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cswf\" (UniqueName: \"kubernetes.io/projected/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-api-access-2cswf\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.295946 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.295559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5c4c891-bee5-45f8-8334-7607a36f95ff-crio-socket\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.295946 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.295642 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.295946 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.295687 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5c4c891-bee5-45f8-8334-7607a36f95ff-data-volume\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.396521 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.396672 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cswf\" (UniqueName: \"kubernetes.io/projected/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-api-access-2cswf\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.396672 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5c4c891-bee5-45f8-8334-7607a36f95ff-crio-socket\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.396672 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.396600 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:29.396672 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.396672 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5c4c891-bee5-45f8-8334-7607a36f95ff-data-volume\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.396672 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.396671 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:29.896650531 +0000 UTC m=+41.356452808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : configmap "kube-rbac-proxy" not found Aug 05 16:23:29.396940 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.396856 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:29.396940 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.396924 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:29.896905761 +0000 UTC m=+41.356708019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:29.397012 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5c4c891-bee5-45f8-8334-7607a36f95ff-data-volume\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.397012 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.396985 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5c4c891-bee5-45f8-8334-7607a36f95ff-crio-socket\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.417460 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.417396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cswf\" (UniqueName: \"kubernetes.io/projected/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-api-access-2cswf\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.624287 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.624253 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl"] Aug 05 16:23:29.646369 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.646334 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl"] Aug 05 16:23:29.646488 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.646433 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" Aug 05 16:23:29.648462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.648437 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 05 16:23:29.648563 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.648482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 05 16:23:29.648604 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.648590 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gbq8f\"" Aug 05 16:23:29.800683 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.800590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswvq\" (UniqueName: \"kubernetes.io/projected/86b0c9ea-bc97-4790-8a68-f8e19c07fa75-kube-api-access-cswvq\") pod \"migrator-68bc745d87-pgbnl\" (UID: \"86b0c9ea-bc97-4790-8a68-f8e19c07fa75\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" Aug 05 16:23:29.901314 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.901279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswvq\" (UniqueName: \"kubernetes.io/projected/86b0c9ea-bc97-4790-8a68-f8e19c07fa75-kube-api-access-cswvq\") pod \"migrator-68bc745d87-pgbnl\" (UID: \"86b0c9ea-bc97-4790-8a68-f8e19c07fa75\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" Aug 05 16:23:29.901439 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.901333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.901491 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.901436 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:29.901521 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.901497 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.901475813 +0000 UTC m=+42.361278075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:29.901586 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.901571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:29.901660 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.901648 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:29.901701 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:29.901692 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:30.901680234 +0000 UTC m=+42.361482492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : configmap "kube-rbac-proxy" not found Aug 05 16:23:29.908858 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.908837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswvq\" (UniqueName: \"kubernetes.io/projected/86b0c9ea-bc97-4790-8a68-f8e19c07fa75-kube-api-access-cswvq\") pod \"migrator-68bc745d87-pgbnl\" (UID: \"86b0c9ea-bc97-4790-8a68-f8e19c07fa75\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" Aug 05 16:23:29.954795 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:29.954770 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" Aug 05 16:23:30.093119 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.093077 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl"] Aug 05 16:23:30.096159 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:30.096134 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b0c9ea_bc97_4790_8a68_f8e19c07fa75.slice/crio-8ed92840258b31aab815d758cc0286315adaaa0dd17ad0052b260792cbdfb416 WatchSource:0}: Error finding container 8ed92840258b31aab815d758cc0286315adaaa0dd17ad0052b260792cbdfb416: Status 404 returned error can't find the container with id 8ed92840258b31aab815d758cc0286315adaaa0dd17ad0052b260792cbdfb416 Aug 05 16:23:30.289378 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.289338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" event={"ID":"86b0c9ea-bc97-4790-8a68-f8e19c07fa75","Type":"ContainerStarted","Data":"8ed92840258b31aab815d758cc0286315adaaa0dd17ad0052b260792cbdfb416"} Aug 05 16:23:30.708664 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.708623 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:30.709050 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.708778 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:30.709050 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.708845 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls podName:62cc78cf-1569-4119-9ad6-5fe13239a970 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:38.708828365 +0000 UTC m=+50.168630643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls") pod "dns-default-dht42" (UID: "62cc78cf-1569-4119-9ad6-5fe13239a970") : secret "dns-default-metrics-tls" not found Aug 05 16:23:30.809328 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.809271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:30.809553 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.809426 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 16:23:30.809553 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.809450 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8566dd4b4b-fzhgr: secret "image-registry-tls" not found Aug 05 16:23:30.809553 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.809523 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls podName:b17f2c80-c696-4cc8-bd0a-a231d3c1f203 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:38.809504687 +0000 UTC m=+50.269306968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls") pod "image-registry-8566dd4b4b-fzhgr" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203") : secret "image-registry-tls" not found Aug 05 16:23:30.907669 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.907633 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrpwk_6676a50f-361b-49f4-b68b-33549816ce4a/dns-node-resolver/0.log" Aug 05 16:23:30.909900 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.909876 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:30.910036 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:30.909936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:30.910036 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.910000 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:30.910127 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.910067 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:32.910046794 +0000 UTC m=+44.369849069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : configmap "kube-rbac-proxy" not found Aug 05 16:23:30.910127 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.910109 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:30.910233 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:30.910169 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:32.910155708 +0000 UTC m=+44.369957966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:31.707499 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:31.707468 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmlcr_cc776abf-676f-43e7-b4b5-7c68a29a7db7/node-ca/0.log" Aug 05 16:23:32.296077 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:32.295996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" event={"ID":"86b0c9ea-bc97-4790-8a68-f8e19c07fa75","Type":"ContainerStarted","Data":"d7d4036fcc4dc72e97f5aba111bcb31356aee6261377495807e620d3559fcfa1"} Aug 05 16:23:32.296077 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:32.296061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" event={"ID":"86b0c9ea-bc97-4790-8a68-f8e19c07fa75","Type":"ContainerStarted","Data":"2d289f54f7a88fd989a7367742b5d095e8f0265dd65cf6815f5c459d3c5485a4"} Aug 05 16:23:32.313174 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:32.313049 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-pgbnl" podStartSLOduration=1.571336007 podStartE2EDuration="3.313028456s" totalImagesPullingTime="1.741692449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:32.312738929 +0000 UTC m=+43.772541211" watchObservedRunningTime="2026-08-05 16:23:32.313028456 +0000 UTC m=+43.772830738" Aug 05 16:23:32.508590 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:32.508535 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vv5rs_f0757141-204c-4f0d-b1af-5c515b125c42/serve-healthcheck-canary/0.log" Aug 05 16:23:32.926725 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:32.926686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:32.926905 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:32.926748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:32.926905 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:32.926818 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:32.926905 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:32.926892 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:36.926876206 +0000 UTC m=+48.386678464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : configmap "kube-rbac-proxy" not found Aug 05 16:23:32.927018 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:32.926907 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:32.927018 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:32.927011 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:36.92699866 +0000 UTC m=+48.386800919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:36.955397 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:36.955360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:36.955825 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:36.955420 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:36.955825 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:36.955489 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:23:36.955825 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:36.955521 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:36.955825 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:36.955567 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:44.955551216 +0000 UTC m=+56.415353478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : configmap "kube-rbac-proxy" not found Aug 05 16:23:36.955825 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:36.955581 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:23:44.95557455 +0000 UTC m=+56.415376808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:38.768418 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:38.768386 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:38.768797 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:38.768503 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:23:38.768797 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:38.768559 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls podName:62cc78cf-1569-4119-9ad6-5fe13239a970 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:54.768542728 +0000 UTC m=+66.228344987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls") pod "dns-default-dht42" (UID: "62cc78cf-1569-4119-9ad6-5fe13239a970") : secret "dns-default-metrics-tls" not found Aug 05 16:23:38.868872 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:38.868844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") pod \"image-registry-8566dd4b4b-fzhgr\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:38.869013 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:38.868946 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 16:23:38.869013 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:38.868957 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8566dd4b4b-fzhgr: secret "image-registry-tls" not found Aug 05 16:23:38.869013 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:38.869002 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls podName:b17f2c80-c696-4cc8-bd0a-a231d3c1f203 nodeName:}" failed. No retries permitted until 2026-08-05 16:23:54.868991361 +0000 UTC m=+66.328793619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls") pod "image-registry-8566dd4b4b-fzhgr" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203") : secret "image-registry-tls" not found Aug 05 16:23:45.013221 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:45.013192 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:45.013679 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:45.013240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:45.013679 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:45.013338 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:23:45.013679 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:45.013389 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls podName:d5c4c891-bee5-45f8-8334-7607a36f95ff nodeName:}" failed. No retries permitted until 2026-08-05 16:24:01.013373666 +0000 UTC m=+72.473175926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9pmrz" (UID: "d5c4c891-bee5-45f8-8334-7607a36f95ff") : secret "insights-runtime-extractor-tls" not found Aug 05 16:23:45.013851 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:45.013814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5c4c891-bee5-45f8-8334-7607a36f95ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:23:48.257652 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:48.257624 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8hsb7" Aug 05 16:23:51.057046 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.057016 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8566dd4b4b-fzhgr"] Aug 05 16:23:51.057492 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:23:51.057232 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" podUID="b17f2c80-c696-4cc8-bd0a-a231d3c1f203" Aug 05 16:23:51.076529 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.076505 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6"] Aug 05 16:23:51.078858 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.078843 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.080229 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.080208 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795"] Aug 05 16:23:51.080442 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.080420 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 05 16:23:51.080746 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.080725 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 05 16:23:51.080826 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.080765 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 05 16:23:51.080826 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.080809 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 05 16:23:51.081831 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.081814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.084387 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.084367 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 05 16:23:51.084491 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.084411 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 05 16:23:51.084555 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.084493 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 05 16:23:51.084684 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.084669 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 05 16:23:51.089713 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.089693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6"] Aug 05 16:23:51.099701 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.099685 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795"] Aug 05 16:23:51.161063 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.160995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-klusterlet-config\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.161063 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-hub\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.161305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfx6g\" (UniqueName: \"kubernetes.io/projected/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-kube-api-access-qfx6g\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.161305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.161305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-tmp\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.161305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161254 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmcj\" (UniqueName: \"kubernetes.io/projected/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-kube-api-access-fqmcj\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.161305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.161514 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-ca\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.161514 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.161354 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262027 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.261986 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-hub\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262194 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262050 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfx6g\" (UniqueName: \"kubernetes.io/projected/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-kube-api-access-qfx6g\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.262194 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262194 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-tmp\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.262319 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmcj\" (UniqueName: \"kubernetes.io/projected/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-kube-api-access-fqmcj\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262319 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-ca\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.262414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-klusterlet-config\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.262533 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.262466 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-tmp\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.263264 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.263235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.264681 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.264654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-hub\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.264811 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.264768 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.265006 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.264987 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-klusterlet-config\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.265006 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.265000 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-ca\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.265138 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.265102 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.275469 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.275442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmcj\" (UniqueName: \"kubernetes.io/projected/f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c-kube-api-access-fqmcj\") pod \"cluster-proxy-proxy-agent-6fdc688b5d-q9795\" (UID: \"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.275558 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.275545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfx6g\" (UniqueName: \"kubernetes.io/projected/6bfb3c63-f5d7-4039-8761-11e42bc17fc9-kube-api-access-qfx6g\") pod \"klusterlet-addon-workmgr-659d5b6d44-rwgh6\" (UID: \"6bfb3c63-f5d7-4039-8761-11e42bc17fc9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.337507 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.337488 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:51.342901 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.342884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:51.362860 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.362838 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-image-registry-private-configuration\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.362942 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.362882 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-ca-trust-extracted\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.362942 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.362908 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-certificates\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.363012 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.362950 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-bound-sa-token\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.363012 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.362976 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-installation-pull-secrets\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.363104 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.363009 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnpns\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-kube-api-access-rnpns\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.363104 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.363051 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-trusted-ca\") pod \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\" (UID: \"b17f2c80-c696-4cc8-bd0a-a231d3c1f203\") " Aug 05 16:23:51.363204 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.363168 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:23:51.363353 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.363323 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:23:51.363464 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.363381 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-ca-trust-extracted\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.363598 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.363576 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:23:51.364992 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.364965 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:23:51.365065 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.365043 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:23:51.365147 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.365107 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-kube-api-access-rnpns" (OuterVolumeSpecName: "kube-api-access-rnpns") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "kube-api-access-rnpns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:23:51.365147 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.365082 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b17f2c80-c696-4cc8-bd0a-a231d3c1f203" (UID: "b17f2c80-c696-4cc8-bd0a-a231d3c1f203"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:23:51.388838 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.388813 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:51.402736 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.402718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" Aug 05 16:23:51.467572 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.467534 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-bound-sa-token\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.467572 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.467572 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-installation-pull-secrets\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.467788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.467589 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnpns\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-kube-api-access-rnpns\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.467788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.467603 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-trusted-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.467788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.467617 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-image-registry-private-configuration\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.467788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.467631 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-certificates\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:51.540786 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.540757 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6"] Aug 05 16:23:51.545208 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:51.545184 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfb3c63_f5d7_4039_8761_11e42bc17fc9.slice/crio-c38ea373f00ac273a3406be5009fc4357ac5f1268f10d72f3c34162a51517c0c WatchSource:0}: Error finding container c38ea373f00ac273a3406be5009fc4357ac5f1268f10d72f3c34162a51517c0c: Status 404 returned error can't find the container with id c38ea373f00ac273a3406be5009fc4357ac5f1268f10d72f3c34162a51517c0c Aug 05 16:23:51.555947 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:51.555921 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795"] Aug 05 16:23:51.558492 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:51.558473 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a6ea80_3b50_4d2a_aab9_22d5251d0c8c.slice/crio-ca429b33eac4a09dd16c950dae1945ba8fc408436cb2440d6ab1cd1386f19fff WatchSource:0}: Error finding container ca429b33eac4a09dd16c950dae1945ba8fc408436cb2440d6ab1cd1386f19fff: Status 404 returned error can't find the container with id ca429b33eac4a09dd16c950dae1945ba8fc408436cb2440d6ab1cd1386f19fff Aug 05 16:23:52.341799 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:52.341758 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" event={"ID":"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c","Type":"ContainerStarted","Data":"ca429b33eac4a09dd16c950dae1945ba8fc408436cb2440d6ab1cd1386f19fff"} Aug 05 16:23:52.343263 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:52.343228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" event={"ID":"6bfb3c63-f5d7-4039-8761-11e42bc17fc9","Type":"ContainerStarted","Data":"c38ea373f00ac273a3406be5009fc4357ac5f1268f10d72f3c34162a51517c0c"} Aug 05 16:23:52.343377 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:52.343242 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8566dd4b4b-fzhgr" Aug 05 16:23:52.383340 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:52.383311 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8566dd4b4b-fzhgr"] Aug 05 16:23:52.416128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:52.416054 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8566dd4b4b-fzhgr"] Aug 05 16:23:52.474491 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:52.474462 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b17f2c80-c696-4cc8-bd0a-a231d3c1f203-registry-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:23:53.011324 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.011290 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17f2c80-c696-4cc8-bd0a-a231d3c1f203" path="/var/lib/kubelet/pods/b17f2c80-c696-4cc8-bd0a-a231d3c1f203/volumes" Aug 05 16:23:53.785819 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.785782 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:53.786284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.785871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:53.787904 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.787878 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 16:23:53.788052 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.787931 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 16:23:53.799025 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.799001 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e464b10-9c26-458f-a3e6-5effcba181f5-original-pull-secret\") pod \"global-pull-secret-syncer-kjwrw\" (UID: \"5e464b10-9c26-458f-a3e6-5effcba181f5\") " pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:53.799162 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.799012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e965746d-d914-436f-b115-0d3ebcc71e6d-metrics-certs\") pod \"network-metrics-daemon-qs6sj\" (UID: \"e965746d-d914-436f-b115-0d3ebcc71e6d\") " pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:53.886934 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.886633 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:53.888727 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.888690 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 16:23:53.899031 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.899008 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 16:23:53.911182 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:53.911137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mvj\" (UniqueName: \"kubernetes.io/projected/68255723-2c55-4175-a70b-b337c49e9f09-kube-api-access-f9mvj\") pod \"network-check-target-ctr2g\" (UID: \"68255723-2c55-4175-a70b-b337c49e9f09\") " pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:54.020466 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.020432 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t7dww\"" Aug 05 16:23:54.025028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.025006 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kjwrw" Aug 05 16:23:54.029144 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.029121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:23:54.032232 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.031953 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zm8j\"" Aug 05 16:23:54.041126 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.041058 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs6sj" Aug 05 16:23:54.794032 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.793791 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:54.796954 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:54.796895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62cc78cf-1569-4119-9ad6-5fe13239a970-metrics-tls\") pod \"dns-default-dht42\" (UID: \"62cc78cf-1569-4119-9ad6-5fe13239a970\") " pod="openshift-dns/dns-default-dht42" Aug 05 16:23:55.073833 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:55.073757 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w85qf\"" Aug 05 16:23:55.082385 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:55.082355 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dht42" Aug 05 16:23:57.830898 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:57.830051 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kjwrw"] Aug 05 16:23:57.835632 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:57.835610 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e464b10_9c26_458f_a3e6_5effcba181f5.slice/crio-075a96ace4378abcf4dba515a45d6be8efcf0edb20cae17bb821a739237e5a9e WatchSource:0}: Error finding container 075a96ace4378abcf4dba515a45d6be8efcf0edb20cae17bb821a739237e5a9e: Status 404 returned error can't find the container with id 075a96ace4378abcf4dba515a45d6be8efcf0edb20cae17bb821a739237e5a9e Aug 05 16:23:57.879849 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:57.879758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qs6sj"] Aug 05 16:23:57.885531 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:57.885494 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode965746d_d914_436f_b115_0d3ebcc71e6d.slice/crio-41338f5f0dffd5840cfcf001816b0f409f5b5ee466e48a4d5f4d8313f0e33639 WatchSource:0}: Error finding container 41338f5f0dffd5840cfcf001816b0f409f5b5ee466e48a4d5f4d8313f0e33639: Status 404 returned error can't find the container with id 41338f5f0dffd5840cfcf001816b0f409f5b5ee466e48a4d5f4d8313f0e33639 Aug 05 16:23:57.894824 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:57.894749 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ctr2g"] Aug 05 16:23:57.898277 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:57.898250 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68255723_2c55_4175_a70b_b337c49e9f09.slice/crio-bbac2f10e5afecafba05d5e6a5eb39237a95ad269d35488425fb76e875e9a753 WatchSource:0}: Error finding container bbac2f10e5afecafba05d5e6a5eb39237a95ad269d35488425fb76e875e9a753: Status 404 returned error can't find the container with id bbac2f10e5afecafba05d5e6a5eb39237a95ad269d35488425fb76e875e9a753 Aug 05 16:23:57.921227 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:57.921142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dht42"] Aug 05 16:23:57.926460 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:23:57.926427 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cc78cf_1569_4119_9ad6_5fe13239a970.slice/crio-7e73e28960dcb443bd7dea15aa07d486e45adac431c8551c82b5248e2e4d296e WatchSource:0}: Error finding container 7e73e28960dcb443bd7dea15aa07d486e45adac431c8551c82b5248e2e4d296e: Status 404 returned error can't find the container with id 7e73e28960dcb443bd7dea15aa07d486e45adac431c8551c82b5248e2e4d296e Aug 05 16:23:58.361771 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.361739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dht42" event={"ID":"62cc78cf-1569-4119-9ad6-5fe13239a970","Type":"ContainerStarted","Data":"7e73e28960dcb443bd7dea15aa07d486e45adac431c8551c82b5248e2e4d296e"} Aug 05 16:23:58.363184 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.363148 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs6sj" event={"ID":"e965746d-d914-436f-b115-0d3ebcc71e6d","Type":"ContainerStarted","Data":"41338f5f0dffd5840cfcf001816b0f409f5b5ee466e48a4d5f4d8313f0e33639"} Aug 05 16:23:58.364483 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.364454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kjwrw" event={"ID":"5e464b10-9c26-458f-a3e6-5effcba181f5","Type":"ContainerStarted","Data":"075a96ace4378abcf4dba515a45d6be8efcf0edb20cae17bb821a739237e5a9e"} Aug 05 16:23:58.366689 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.366663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" event={"ID":"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c","Type":"ContainerStarted","Data":"3a83fe2a6aff1c4a19fb2d9c038d1cd806e48bee5e677aefcf6147e599fd925d"} Aug 05 16:23:58.366803 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.366699 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" event={"ID":"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c","Type":"ContainerStarted","Data":"d7ef5f6aecb9de7b60691099d657c64431ab333756c31a637b4c25770b8cb45e"} Aug 05 16:23:58.366803 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.366717 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" event={"ID":"f6a6ea80-3b50-4d2a-aab9-22d5251d0c8c","Type":"ContainerStarted","Data":"182184253cc57fb9631b0de6c952040b758f68ac1f78ff15bc5ecae42ce83f96"} Aug 05 16:23:58.368315 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.368288 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" event={"ID":"6bfb3c63-f5d7-4039-8761-11e42bc17fc9","Type":"ContainerStarted","Data":"36ecbd357d2321ec42489b61fb9a0330356c3e5137851b23db91fe914ed9d0bb"} Aug 05 16:23:58.368869 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.368844 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:58.370303 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.370281 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ctr2g" event={"ID":"68255723-2c55-4175-a70b-b337c49e9f09","Type":"ContainerStarted","Data":"bbac2f10e5afecafba05d5e6a5eb39237a95ad269d35488425fb76e875e9a753"} Aug 05 16:23:58.370478 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.370461 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" Aug 05 16:23:58.387452 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.383658 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fdc688b5d-q9795" podStartSLOduration=1.221756645 podStartE2EDuration="7.383634404s" totalImagesPullingTime="6.161877759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:58.383365736 +0000 UTC m=+69.843168018" watchObservedRunningTime="2026-08-05 16:23:58.383634404 +0000 UTC m=+69.843436685" Aug 05 16:23:58.398639 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:23:58.398026 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-659d5b6d44-rwgh6" podStartSLOduration=1.204350128 podStartE2EDuration="7.398008606s" totalImagesPullingTime="6.193658478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:23:58.397286787 +0000 UTC m=+69.857089068" watchObservedRunningTime="2026-08-05 16:23:58.398008606 +0000 UTC m=+69.857810888" Aug 05 16:24:01.040827 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:01.040787 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:24:01.043502 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:01.043475 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5c4c891-bee5-45f8-8334-7607a36f95ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9pmrz\" (UID: \"d5c4c891-bee5-45f8-8334-7607a36f95ff\") " pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:24:01.295887 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:01.295799 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-422vm\"" Aug 05 16:24:01.303667 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:01.303640 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9pmrz" Aug 05 16:24:02.670340 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:02.670177 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9pmrz"] Aug 05 16:24:02.674141 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:24:02.674116 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c4c891_bee5_45f8_8334_7607a36f95ff.slice/crio-6737d74e7aa077c09cdea3ea61337e64366125d24fd1831de03fec52c04e07c5 WatchSource:0}: Error finding container 6737d74e7aa077c09cdea3ea61337e64366125d24fd1831de03fec52c04e07c5: Status 404 returned error can't find the container with id 6737d74e7aa077c09cdea3ea61337e64366125d24fd1831de03fec52c04e07c5 Aug 05 16:24:03.389052 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.389013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kjwrw" event={"ID":"5e464b10-9c26-458f-a3e6-5effcba181f5","Type":"ContainerStarted","Data":"b4b48d25c396625898de8d47f38650f05e673ff6baf9b5432fe28d917a5a408a"} Aug 05 16:24:03.390558 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.390529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ctr2g" event={"ID":"68255723-2c55-4175-a70b-b337c49e9f09","Type":"ContainerStarted","Data":"378e13b1f1ed73afd358b638d59eb9a3719c43eab1782664b946a25e458676c9"} Aug 05 16:24:03.390694 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.390630 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:24:03.392508 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.392484 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dht42" event={"ID":"62cc78cf-1569-4119-9ad6-5fe13239a970","Type":"ContainerStarted","Data":"951c0fcecf10eb81776786aa975e093c1bbc806e7fc5831499cc64bd14f28f13"} Aug 05 16:24:03.392632 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.392539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dht42" event={"ID":"62cc78cf-1569-4119-9ad6-5fe13239a970","Type":"ContainerStarted","Data":"4702f5b65f86300aaf4c0e1e3c37d40e4d36e46592146d3cd8a87269d0095454"} Aug 05 16:24:03.392632 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.392569 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dht42" Aug 05 16:24:03.394039 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.394018 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pmrz" event={"ID":"d5c4c891-bee5-45f8-8334-7607a36f95ff","Type":"ContainerStarted","Data":"586993dd288f13330c6c9106bd287aca8409b206a089fa8f00942c2f56027fa6"} Aug 05 16:24:03.394162 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.394056 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pmrz" event={"ID":"d5c4c891-bee5-45f8-8334-7607a36f95ff","Type":"ContainerStarted","Data":"6737d74e7aa077c09cdea3ea61337e64366125d24fd1831de03fec52c04e07c5"} Aug 05 16:24:03.395827 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.395795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs6sj" event={"ID":"e965746d-d914-436f-b115-0d3ebcc71e6d","Type":"ContainerStarted","Data":"87cb49133c48f83c9687f502e37113f47e3161e357242ea5241cb7f7461cfab1"} Aug 05 16:24:03.395938 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.395834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs6sj" event={"ID":"e965746d-d914-436f-b115-0d3ebcc71e6d","Type":"ContainerStarted","Data":"0d51ba8393d2dbd57a14ce7705f4c362ab4d244867dace37cd52d15f54c7988a"} Aug 05 16:24:03.404378 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.404332 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kjwrw" podStartSLOduration=69.651587617 podStartE2EDuration="1m14.404320336s" totalImagesPullingTime="4.752732719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:03.403566488 +0000 UTC m=+74.863368769" watchObservedRunningTime="2026-08-05 16:24:03.404320336 +0000 UTC m=+74.864122615" Aug 05 16:24:03.419920 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.419687 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dht42" podStartSLOduration=36.821567118 podStartE2EDuration="41.419666988s" totalImagesPullingTime="4.59809987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:03.418616566 +0000 UTC m=+74.878418848" watchObservedRunningTime="2026-08-05 16:24:03.419666988 +0000 UTC m=+74.879469270" Aug 05 16:24:03.440446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.440389 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ctr2g" podStartSLOduration=69.777759184 podStartE2EDuration="1m14.44037075s" totalImagesPullingTime="4.662611566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:03.439651748 +0000 UTC m=+74.899454022" watchObservedRunningTime="2026-08-05 16:24:03.44037075 +0000 UTC m=+74.900173030" Aug 05 16:24:03.455446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:03.455401 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qs6sj" podStartSLOduration=69.818066963 podStartE2EDuration="1m14.455386044s" totalImagesPullingTime="4.637319081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:03.454120208 +0000 UTC m=+74.913922487" watchObservedRunningTime="2026-08-05 16:24:03.455386044 +0000 UTC m=+74.915188326" Aug 05 16:24:04.400586 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:04.400546 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pmrz" event={"ID":"d5c4c891-bee5-45f8-8334-7607a36f95ff","Type":"ContainerStarted","Data":"08137de0a6dace68b8800612a10bfb4247a010b4e5495cf92fbf3ce3dd9f11d8"} Aug 05 16:24:05.405636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:05.405600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9pmrz" event={"ID":"d5c4c891-bee5-45f8-8334-7607a36f95ff","Type":"ContainerStarted","Data":"219a0d10c5269b9c33d0ea4dd97ebb2ae7b5f0cc8e1626465622a154ef435b55"} Aug 05 16:24:05.425302 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:05.425250 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9pmrz" podStartSLOduration=34.351959656 podStartE2EDuration="36.425231366s" totalImagesPullingTime="2.07327171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:23:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:05.424388153 +0000 UTC m=+76.884190434" watchObservedRunningTime="2026-08-05 16:24:05.425231366 +0000 UTC m=+76.885033646" Aug 05 16:24:06.148381 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.148344 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zpn8p"] Aug 05 16:24:06.152419 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.152397 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.154481 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.154456 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 05 16:24:06.154589 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.154499 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 05 16:24:06.154656 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.154597 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 05 16:24:06.154707 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.154673 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pxz2s\"" Aug 05 16:24:06.154755 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.154737 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 05 16:24:06.155245 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.155225 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 05 16:24:06.155331 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.155273 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 05 16:24:06.283820 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.283782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-tls\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.283820 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.283822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-metrics-client-ca\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.283850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.283869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-sys\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.283889 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-textfile\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.283958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-wtmp\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.284000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6695\" (UniqueName: \"kubernetes.io/projected/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-kube-api-access-v6695\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.284024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-root\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.284280 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.284051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384427 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384427 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-tls\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-metrics-client-ca\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-sys\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384516 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-textfile\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-wtmp\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6695\" (UniqueName: \"kubernetes.io/projected/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-kube-api-access-v6695\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-sys\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-root\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.384642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-root\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.385044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-wtmp\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.385044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.384999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-textfile\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.385154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.385082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.385154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.385113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-metrics-client-ca\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.386699 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.386682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.386832 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.386804 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-node-exporter-tls\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.392472 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.392452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6695\" (UniqueName: \"kubernetes.io/projected/ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d-kube-api-access-v6695\") pod \"node-exporter-zpn8p\" (UID: \"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d\") " pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.460850 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:06.460763 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zpn8p" Aug 05 16:24:06.473568 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:24:06.473536 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead2f9ba_7a97_45dc_9dd1_11982a5e3a7d.slice/crio-e88029fe8e8e47cb2b2347956ecb7153c90a6306c2d6aaa9e32aef0883260f65 WatchSource:0}: Error finding container e88029fe8e8e47cb2b2347956ecb7153c90a6306c2d6aaa9e32aef0883260f65: Status 404 returned error can't find the container with id e88029fe8e8e47cb2b2347956ecb7153c90a6306c2d6aaa9e32aef0883260f65 Aug 05 16:24:07.414526 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:07.414486 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpn8p" event={"ID":"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d","Type":"ContainerStarted","Data":"e88029fe8e8e47cb2b2347956ecb7153c90a6306c2d6aaa9e32aef0883260f65"} Aug 05 16:24:08.419028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:08.418987 2567 generic.go:356] "Generic (PLEG): container finished" podID="ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d" containerID="9f33c986a612176afd36d89a71906ca53e341f63253056e540b8088b6089af41" exitCode=0 Aug 05 16:24:08.419436 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:08.419063 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpn8p" event={"ID":"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d","Type":"ContainerDied","Data":"9f33c986a612176afd36d89a71906ca53e341f63253056e540b8088b6089af41"} Aug 05 16:24:09.424613 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:09.424571 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpn8p" event={"ID":"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d","Type":"ContainerStarted","Data":"e9947363ae9ee9c377f4c67ed683f6731139e21e1bcda667a92396d7733353e7"} Aug 05 16:24:09.424999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:09.424627 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zpn8p" event={"ID":"ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d","Type":"ContainerStarted","Data":"6067aeacfca0d7506a1211001bc1d8e5e8529eda8bc8f62c06489b42c7df4613"} Aug 05 16:24:09.440921 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:09.440865 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zpn8p" podStartSLOduration=2.035565594 podStartE2EDuration="3.440846215s" totalImagesPullingTime="1.340807907s" totalInitContainerRuntime="64.472714ms" isStatefulPod=true podCreationTimestamp="2026-08-05 16:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:09.440585267 +0000 UTC m=+80.900387544" watchObservedRunningTime="2026-08-05 16:24:09.440846215 +0000 UTC m=+80.900648473" Aug 05 16:24:13.403464 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:13.403338 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dht42" Aug 05 16:24:24.493365 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.493324 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dcc687cd-4rbr4"] Aug 05 16:24:24.497899 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.497882 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.499938 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.499902 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 05 16:24:24.499938 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.499904 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 05 16:24:24.500237 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.500220 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 05 16:24:24.500337 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.500256 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 05 16:24:24.500337 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.500329 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kwgg\"" Aug 05 16:24:24.500476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.500339 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 05 16:24:24.500476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.500348 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 05 16:24:24.500476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.500403 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 05 16:24:24.506697 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.506670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcc687cd-4rbr4"] Aug 05 16:24:24.610407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.610373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-oauth-config\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.610407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.610414 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-console-config\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.610611 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.610435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-oauth-serving-cert\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.610611 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.610531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-service-ca\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.610611 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.610598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-serving-cert\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.610741 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.610618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwxs\" (UniqueName: \"kubernetes.io/projected/556e168b-0f4e-46a4-ad91-2b942932374e-kube-api-access-cfwxs\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.711382 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.711348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-serving-cert\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.711382 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.711383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwxs\" (UniqueName: \"kubernetes.io/projected/556e168b-0f4e-46a4-ad91-2b942932374e-kube-api-access-cfwxs\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.711520 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.711414 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-oauth-config\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.711520 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.711447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-console-config\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.711520 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.711477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-oauth-serving-cert\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.711520 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.711507 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-service-ca\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.712254 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.712225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-console-config\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.712362 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.712259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-service-ca\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.712409 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.712369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-oauth-serving-cert\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.713826 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.713802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-serving-cert\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.713914 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.713836 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-oauth-config\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.722704 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.722681 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwxs\" (UniqueName: \"kubernetes.io/projected/556e168b-0f4e-46a4-ad91-2b942932374e-kube-api-access-cfwxs\") pod \"console-5dcc687cd-4rbr4\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.807807 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.807718 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:24.949181 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:24.949137 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcc687cd-4rbr4"] Aug 05 16:24:24.951591 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:24:24.951565 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556e168b_0f4e_46a4_ad91_2b942932374e.slice/crio-ef02944d8f06689caacd2df8a8a4f06feebd80ec5688fabcd63668147191aa26 WatchSource:0}: Error finding container ef02944d8f06689caacd2df8a8a4f06feebd80ec5688fabcd63668147191aa26: Status 404 returned error can't find the container with id ef02944d8f06689caacd2df8a8a4f06feebd80ec5688fabcd63668147191aa26 Aug 05 16:24:25.478963 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:25.478922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc687cd-4rbr4" event={"ID":"556e168b-0f4e-46a4-ad91-2b942932374e","Type":"ContainerStarted","Data":"ef02944d8f06689caacd2df8a8a4f06feebd80ec5688fabcd63668147191aa26"} Aug 05 16:24:28.496829 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:28.496732 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc687cd-4rbr4" event={"ID":"556e168b-0f4e-46a4-ad91-2b942932374e","Type":"ContainerStarted","Data":"28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2"} Aug 05 16:24:28.515275 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:28.515202 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dcc687cd-4rbr4" podStartSLOduration=1.34052031 podStartE2EDuration="4.515184872s" totalImagesPullingTime="3.174664562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:24:28.514732202 +0000 UTC m=+99.974534483" watchObservedRunningTime="2026-08-05 16:24:28.515184872 +0000 UTC m=+99.974987152" Aug 05 16:24:34.403294 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:34.403267 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ctr2g" Aug 05 16:24:34.808730 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:34.808645 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:34.808730 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:34.808705 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:34.813520 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:34.813496 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:35.524016 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:35.523983 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:24:43.710996 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:24:43.710961 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcc687cd-4rbr4"] Aug 05 16:25:08.729870 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:08.729825 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dcc687cd-4rbr4" podUID="556e168b-0f4e-46a4-ad91-2b942932374e" containerName="console" containerID="cri-o://28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2" gracePeriod=15 Aug 05 16:25:08.964068 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:08.964044 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:25:09.026415 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026349 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-oauth-config\") pod \"556e168b-0f4e-46a4-ad91-2b942932374e\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " Aug 05 16:25:09.026415 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026376 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-serving-cert\") pod \"556e168b-0f4e-46a4-ad91-2b942932374e\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " Aug 05 16:25:09.026415 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026396 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-console-config\") pod \"556e168b-0f4e-46a4-ad91-2b942932374e\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " Aug 05 16:25:09.026415 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026416 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-oauth-serving-cert\") pod \"556e168b-0f4e-46a4-ad91-2b942932374e\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " Aug 05 16:25:09.026710 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026431 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-service-ca\") pod \"556e168b-0f4e-46a4-ad91-2b942932374e\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " Aug 05 16:25:09.026710 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026466 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwxs\" (UniqueName: \"kubernetes.io/projected/556e168b-0f4e-46a4-ad91-2b942932374e-kube-api-access-cfwxs\") pod \"556e168b-0f4e-46a4-ad91-2b942932374e\" (UID: \"556e168b-0f4e-46a4-ad91-2b942932374e\") " Aug 05 16:25:09.026809 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026787 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-console-config" (OuterVolumeSpecName: "console-config") pod "556e168b-0f4e-46a4-ad91-2b942932374e" (UID: "556e168b-0f4e-46a4-ad91-2b942932374e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:25:09.026856 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026795 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "556e168b-0f4e-46a4-ad91-2b942932374e" (UID: "556e168b-0f4e-46a4-ad91-2b942932374e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:25:09.026856 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.026805 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-service-ca" (OuterVolumeSpecName: "service-ca") pod "556e168b-0f4e-46a4-ad91-2b942932374e" (UID: "556e168b-0f4e-46a4-ad91-2b942932374e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:25:09.028620 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.028598 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "556e168b-0f4e-46a4-ad91-2b942932374e" (UID: "556e168b-0f4e-46a4-ad91-2b942932374e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:09.028711 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.028645 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "556e168b-0f4e-46a4-ad91-2b942932374e" (UID: "556e168b-0f4e-46a4-ad91-2b942932374e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:25:09.028711 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.028645 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556e168b-0f4e-46a4-ad91-2b942932374e-kube-api-access-cfwxs" (OuterVolumeSpecName: "kube-api-access-cfwxs") pod "556e168b-0f4e-46a4-ad91-2b942932374e" (UID: "556e168b-0f4e-46a4-ad91-2b942932374e"). InnerVolumeSpecName "kube-api-access-cfwxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:25:09.127205 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.127167 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-console-config\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:25:09.127205 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.127197 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-oauth-serving-cert\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:25:09.127205 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.127207 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/556e168b-0f4e-46a4-ad91-2b942932374e-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:25:09.127205 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.127216 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwxs\" (UniqueName: \"kubernetes.io/projected/556e168b-0f4e-46a4-ad91-2b942932374e-kube-api-access-cfwxs\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:25:09.127475 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.127225 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-oauth-config\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:25:09.127475 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.127233 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/556e168b-0f4e-46a4-ad91-2b942932374e-console-serving-cert\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:25:09.629610 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.629572 2567 generic.go:356] "Generic (PLEG): container finished" podID="556e168b-0f4e-46a4-ad91-2b942932374e" containerID="28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2" exitCode=0 Aug 05 16:25:09.629768 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.629620 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc687cd-4rbr4" event={"ID":"556e168b-0f4e-46a4-ad91-2b942932374e","Type":"ContainerDied","Data":"28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2"} Aug 05 16:25:09.629768 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.629627 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc687cd-4rbr4" Aug 05 16:25:09.629768 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.629657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc687cd-4rbr4" event={"ID":"556e168b-0f4e-46a4-ad91-2b942932374e","Type":"ContainerDied","Data":"ef02944d8f06689caacd2df8a8a4f06feebd80ec5688fabcd63668147191aa26"} Aug 05 16:25:09.629768 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.629675 2567 scope.go:117] "RemoveContainer" containerID="28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2" Aug 05 16:25:09.638148 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.638127 2567 scope.go:117] "RemoveContainer" containerID="28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2" Aug 05 16:25:09.638386 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:25:09.638364 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2\": container with ID starting with 28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2 not found: ID does not exist" containerID="28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2" Aug 05 16:25:09.638436 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.638394 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2"} err="failed to get container status \"28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2\": rpc error: code = NotFound desc = could not find container \"28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2\": container with ID starting with 28667b277012b098eee4adcbbca85f027adc00618baaf386f946b7991134aab2 not found: ID does not exist" Aug 05 16:25:09.648920 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.648900 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcc687cd-4rbr4"] Aug 05 16:25:09.652058 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:09.652040 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dcc687cd-4rbr4"] Aug 05 16:25:11.010288 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:25:11.010258 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="556e168b-0f4e-46a4-ad91-2b942932374e" path="/var/lib/kubelet/pods/556e168b-0f4e-46a4-ad91-2b942932374e/volumes" Aug 05 16:26:47.572827 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.572789 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6595c5965c-xpbgf"] Aug 05 16:26:47.573269 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.573035 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="556e168b-0f4e-46a4-ad91-2b942932374e" containerName="console" Aug 05 16:26:47.573269 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.573047 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="556e168b-0f4e-46a4-ad91-2b942932374e" containerName="console" Aug 05 16:26:47.573269 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.573117 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="556e168b-0f4e-46a4-ad91-2b942932374e" containerName="console" Aug 05 16:26:47.575860 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.575844 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.577950 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.577924 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 05 16:26:47.578075 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.578054 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 05 16:26:47.578158 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.578110 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 05 16:26:47.578631 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.578609 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 05 16:26:47.578730 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.578609 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 05 16:26:47.579290 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.579273 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 05 16:26:47.579372 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.579297 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 05 16:26:47.579436 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.579410 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kwgg\"" Aug 05 16:26:47.584433 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.584414 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 05 16:26:47.587342 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.587323 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6595c5965c-xpbgf"] Aug 05 16:26:47.707446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4pr\" (UniqueName: \"kubernetes.io/projected/817874d4-92e5-4c57-80a0-1e8ec74b39c7-kube-api-access-9l4pr\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.707446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707449 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-serving-cert\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.707709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707484 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-oauth-serving-cert\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.707709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-service-ca\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.707709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707577 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-oauth-config\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.707709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-config\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.707709 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.707669 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-trusted-ca-bundle\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808730 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-config\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808730 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808736 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-trusted-ca-bundle\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808979 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4pr\" (UniqueName: \"kubernetes.io/projected/817874d4-92e5-4c57-80a0-1e8ec74b39c7-kube-api-access-9l4pr\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808979 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-serving-cert\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808979 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808833 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-oauth-serving-cert\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808979 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-service-ca\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.808979 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.808924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-oauth-config\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.809677 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.809650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-oauth-serving-cert\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.809806 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.809688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-config\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.809806 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.809645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-service-ca\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.809923 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.809854 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-trusted-ca-bundle\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.811410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.811383 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-oauth-config\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.811522 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.811503 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-serving-cert\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.816797 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.816775 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4pr\" (UniqueName: \"kubernetes.io/projected/817874d4-92e5-4c57-80a0-1e8ec74b39c7-kube-api-access-9l4pr\") pod \"console-6595c5965c-xpbgf\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:47.885014 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:47.884978 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:48.015460 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:48.015426 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6595c5965c-xpbgf"] Aug 05 16:26:48.018365 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:26:48.018334 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817874d4_92e5_4c57_80a0_1e8ec74b39c7.slice/crio-f17aa8163aa80eeaa51d7030bb7f51a4e2b2ac6733035c5e540b9628725584b7 WatchSource:0}: Error finding container f17aa8163aa80eeaa51d7030bb7f51a4e2b2ac6733035c5e540b9628725584b7: Status 404 returned error can't find the container with id f17aa8163aa80eeaa51d7030bb7f51a4e2b2ac6733035c5e540b9628725584b7 Aug 05 16:26:48.946030 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:48.945993 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6595c5965c-xpbgf" event={"ID":"817874d4-92e5-4c57-80a0-1e8ec74b39c7","Type":"ContainerStarted","Data":"213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5"} Aug 05 16:26:48.946531 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:48.946056 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6595c5965c-xpbgf" event={"ID":"817874d4-92e5-4c57-80a0-1e8ec74b39c7","Type":"ContainerStarted","Data":"f17aa8163aa80eeaa51d7030bb7f51a4e2b2ac6733035c5e540b9628725584b7"} Aug 05 16:26:48.964876 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:48.964827 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6595c5965c-xpbgf" podStartSLOduration=1.964810547 podStartE2EDuration="1.964810547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:26:48.963932236 +0000 UTC m=+240.423734517" watchObservedRunningTime="2026-08-05 16:26:48.964810547 +0000 UTC m=+240.424612827" Aug 05 16:26:57.885875 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:57.885836 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:57.886390 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:57.885901 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:57.890758 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:57.890735 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:26:57.978655 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:26:57.978629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:27:48.887237 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:27:48.887193 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:27:48.887721 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:27:48.887252 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:27:48.893780 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:27:48.893758 2567 kubelet.go:1648] "Image garbage collection succeeded" Aug 05 16:28:29.210527 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.210490 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s"] Aug 05 16:28:29.213784 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.213764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.215541 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.215524 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 05 16:28:29.215635 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.215525 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 05 16:28:29.215821 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.215806 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2cxhs\"" Aug 05 16:28:29.222057 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.222038 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s"] Aug 05 16:28:29.309512 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.309475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.309679 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.309524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.309679 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.309594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74n6\" (UniqueName: \"kubernetes.io/projected/e2256518-8fcf-41ef-9e21-ad4767544907-kube-api-access-r74n6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.410663 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.410639 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74n6\" (UniqueName: \"kubernetes.io/projected/e2256518-8fcf-41ef-9e21-ad4767544907-kube-api-access-r74n6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.410829 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.410705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.410829 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.410744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.411602 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.411575 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.411754 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.411579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.418387 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.418359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74n6\" (UniqueName: \"kubernetes.io/projected/e2256518-8fcf-41ef-9e21-ad4767544907-kube-api-access-r74n6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.522266 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.522213 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:29.640954 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.640924 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s"] Aug 05 16:28:29.643742 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:28:29.643716 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2256518_8fcf_41ef_9e21_ad4767544907.slice/crio-6cf58d33a2a8d9c0b3c05d55a2bafe1097dc46afb0e558e35d2098020a80cc0e WatchSource:0}: Error finding container 6cf58d33a2a8d9c0b3c05d55a2bafe1097dc46afb0e558e35d2098020a80cc0e: Status 404 returned error can't find the container with id 6cf58d33a2a8d9c0b3c05d55a2bafe1097dc46afb0e558e35d2098020a80cc0e Aug 05 16:28:29.645605 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:29.645589 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:28:30.259788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:30.259750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" event={"ID":"e2256518-8fcf-41ef-9e21-ad4767544907","Type":"ContainerStarted","Data":"6cf58d33a2a8d9c0b3c05d55a2bafe1097dc46afb0e558e35d2098020a80cc0e"} Aug 05 16:28:36.286017 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:36.285966 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2256518-8fcf-41ef-9e21-ad4767544907" containerID="9d9ea584cc9be22c9178ffd53fe183150ae0710d16aba7f2fc04605118f73909" exitCode=0 Aug 05 16:28:36.286447 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:36.286064 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" event={"ID":"e2256518-8fcf-41ef-9e21-ad4767544907","Type":"ContainerDied","Data":"9d9ea584cc9be22c9178ffd53fe183150ae0710d16aba7f2fc04605118f73909"} Aug 05 16:28:39.299354 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:39.299315 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2256518-8fcf-41ef-9e21-ad4767544907" containerID="e3494b02d8b1c5a0172679d5bf89920f8a6e792fc6a29f6b4c8b3fdca8e1c63e" exitCode=0 Aug 05 16:28:39.299354 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:39.299354 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" event={"ID":"e2256518-8fcf-41ef-9e21-ad4767544907","Type":"ContainerDied","Data":"e3494b02d8b1c5a0172679d5bf89920f8a6e792fc6a29f6b4c8b3fdca8e1c63e"} Aug 05 16:28:46.327438 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:46.327403 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2256518-8fcf-41ef-9e21-ad4767544907" containerID="ac203f11ecb93d1c8fc77cba13dfaea79d7b38e8eafaa4f6fb09d7c76b4b99ae" exitCode=0 Aug 05 16:28:46.327824 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:46.327464 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" event={"ID":"e2256518-8fcf-41ef-9e21-ad4767544907","Type":"ContainerDied","Data":"ac203f11ecb93d1c8fc77cba13dfaea79d7b38e8eafaa4f6fb09d7c76b4b99ae"} Aug 05 16:28:47.450958 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.450934 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:47.543904 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.543869 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-bundle\") pod \"e2256518-8fcf-41ef-9e21-ad4767544907\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " Aug 05 16:28:47.543904 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.543901 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r74n6\" (UniqueName: \"kubernetes.io/projected/e2256518-8fcf-41ef-9e21-ad4767544907-kube-api-access-r74n6\") pod \"e2256518-8fcf-41ef-9e21-ad4767544907\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " Aug 05 16:28:47.544163 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.543945 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-util\") pod \"e2256518-8fcf-41ef-9e21-ad4767544907\" (UID: \"e2256518-8fcf-41ef-9e21-ad4767544907\") " Aug 05 16:28:47.544606 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.544578 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-bundle" (OuterVolumeSpecName: "bundle") pod "e2256518-8fcf-41ef-9e21-ad4767544907" (UID: "e2256518-8fcf-41ef-9e21-ad4767544907"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:28:47.546102 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.546067 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2256518-8fcf-41ef-9e21-ad4767544907-kube-api-access-r74n6" (OuterVolumeSpecName: "kube-api-access-r74n6") pod "e2256518-8fcf-41ef-9e21-ad4767544907" (UID: "e2256518-8fcf-41ef-9e21-ad4767544907"). InnerVolumeSpecName "kube-api-access-r74n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:28:47.548176 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.548151 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-util" (OuterVolumeSpecName: "util") pod "e2256518-8fcf-41ef-9e21-ad4767544907" (UID: "e2256518-8fcf-41ef-9e21-ad4767544907"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:28:47.644529 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.644451 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:28:47.644529 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.644477 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2256518-8fcf-41ef-9e21-ad4767544907-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:28:47.644529 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:47.644487 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74n6\" (UniqueName: \"kubernetes.io/projected/e2256518-8fcf-41ef-9e21-ad4767544907-kube-api-access-r74n6\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:28:48.336297 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:48.336260 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" event={"ID":"e2256518-8fcf-41ef-9e21-ad4767544907","Type":"ContainerDied","Data":"6cf58d33a2a8d9c0b3c05d55a2bafe1097dc46afb0e558e35d2098020a80cc0e"} Aug 05 16:28:48.336297 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:48.336292 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540svm6s" Aug 05 16:28:48.336297 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:48.336306 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf58d33a2a8d9c0b3c05d55a2bafe1097dc46afb0e558e35d2098020a80cc0e" Aug 05 16:28:52.158135 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158100 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8"] Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158430 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="util" Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158446 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="util" Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158457 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="pull" Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158465 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="pull" Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158475 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="extract" Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158483 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="extract" Aug 05 16:28:52.158600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.158537 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2256518-8fcf-41ef-9e21-ad4767544907" containerName="extract" Aug 05 16:28:52.185419 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.185395 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8"] Aug 05 16:28:52.185561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.185481 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.187737 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.187709 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7zwqj\"" Aug 05 16:28:52.187845 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.187709 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 05 16:28:52.187845 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.187797 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:28:52.276546 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.276511 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b58a1f9c-a707-4c6d-985f-bcfaff529ac1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzzr8\" (UID: \"b58a1f9c-a707-4c6d-985f-bcfaff529ac1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.276713 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.276562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkql\" (UniqueName: \"kubernetes.io/projected/b58a1f9c-a707-4c6d-985f-bcfaff529ac1-kube-api-access-2nkql\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzzr8\" (UID: \"b58a1f9c-a707-4c6d-985f-bcfaff529ac1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.377374 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.377341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkql\" (UniqueName: \"kubernetes.io/projected/b58a1f9c-a707-4c6d-985f-bcfaff529ac1-kube-api-access-2nkql\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzzr8\" (UID: \"b58a1f9c-a707-4c6d-985f-bcfaff529ac1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.377526 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.377396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b58a1f9c-a707-4c6d-985f-bcfaff529ac1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzzr8\" (UID: \"b58a1f9c-a707-4c6d-985f-bcfaff529ac1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.377772 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.377757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b58a1f9c-a707-4c6d-985f-bcfaff529ac1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzzr8\" (UID: \"b58a1f9c-a707-4c6d-985f-bcfaff529ac1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.385178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.385151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkql\" (UniqueName: \"kubernetes.io/projected/b58a1f9c-a707-4c6d-985f-bcfaff529ac1-kube-api-access-2nkql\") pod \"cert-manager-operator-controller-manager-55c444bf86-xzzr8\" (UID: \"b58a1f9c-a707-4c6d-985f-bcfaff529ac1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.494729 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.494649 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" Aug 05 16:28:52.661126 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:52.660932 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8"] Aug 05 16:28:52.664013 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:28:52.663986 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58a1f9c_a707_4c6d_985f_bcfaff529ac1.slice/crio-ba676142a9053facbd7fe3e0e5b915ca644d3eb6408c91ab1977dd431ff8eed4 WatchSource:0}: Error finding container ba676142a9053facbd7fe3e0e5b915ca644d3eb6408c91ab1977dd431ff8eed4: Status 404 returned error can't find the container with id ba676142a9053facbd7fe3e0e5b915ca644d3eb6408c91ab1977dd431ff8eed4 Aug 05 16:28:53.355392 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:53.355354 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" event={"ID":"b58a1f9c-a707-4c6d-985f-bcfaff529ac1","Type":"ContainerStarted","Data":"ba676142a9053facbd7fe3e0e5b915ca644d3eb6408c91ab1977dd431ff8eed4"} Aug 05 16:28:56.367788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.367750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" event={"ID":"b58a1f9c-a707-4c6d-985f-bcfaff529ac1","Type":"ContainerStarted","Data":"5fe5c553721a9a99777fd994f73d6b3bb30e783bb483673788aaffad3fb95bc7"} Aug 05 16:28:56.387890 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.387841 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xzzr8" podStartSLOduration=1.244315579 podStartE2EDuration="4.387821972s" totalImagesPullingTime="3.143506393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:28:56.386037275 +0000 UTC m=+367.845839556" watchObservedRunningTime="2026-08-05 16:28:56.387821972 +0000 UTC m=+367.847624253" Aug 05 16:28:56.672951 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.672877 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zt86p"] Aug 05 16:28:56.694589 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.694555 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zt86p"] Aug 05 16:28:56.694741 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.694682 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:56.697128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.697080 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8zkzv\"" Aug 05 16:28:56.697128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.697123 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 05 16:28:56.697321 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.697259 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 05 16:28:56.813966 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.813912 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ghj\" (UniqueName: \"kubernetes.io/projected/78c88b31-b305-4794-bed6-644030bf41ff-kube-api-access-k4ghj\") pod \"cert-manager-webhook-6486ffd997-zt86p\" (UID: \"78c88b31-b305-4794-bed6-644030bf41ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:56.813966 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.813979 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c88b31-b305-4794-bed6-644030bf41ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zt86p\" (UID: \"78c88b31-b305-4794-bed6-644030bf41ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:56.914542 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.914503 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ghj\" (UniqueName: \"kubernetes.io/projected/78c88b31-b305-4794-bed6-644030bf41ff-kube-api-access-k4ghj\") pod \"cert-manager-webhook-6486ffd997-zt86p\" (UID: \"78c88b31-b305-4794-bed6-644030bf41ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:56.914731 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.914572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c88b31-b305-4794-bed6-644030bf41ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zt86p\" (UID: \"78c88b31-b305-4794-bed6-644030bf41ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:56.933469 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.933394 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78c88b31-b305-4794-bed6-644030bf41ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zt86p\" (UID: \"78c88b31-b305-4794-bed6-644030bf41ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:56.935802 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:56.935782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ghj\" (UniqueName: \"kubernetes.io/projected/78c88b31-b305-4794-bed6-644030bf41ff-kube-api-access-k4ghj\") pod \"cert-manager-webhook-6486ffd997-zt86p\" (UID: \"78c88b31-b305-4794-bed6-644030bf41ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:57.003832 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.003788 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:28:57.090642 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.090614 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s"] Aug 05 16:28:57.105850 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.105820 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s"] Aug 05 16:28:57.106030 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.105938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.118942 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.118919 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 05 16:28:57.119340 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.119317 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 05 16:28:57.119434 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.119399 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2cxhs\"" Aug 05 16:28:57.152841 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.152811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zt86p"] Aug 05 16:28:57.156013 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:28:57.155983 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78c88b31_b305_4794_bed6_644030bf41ff.slice/crio-b0f2bfa0730f2881d75ac1faf61185b69da7573dd4569cc48e1663de75f9f650 WatchSource:0}: Error finding container b0f2bfa0730f2881d75ac1faf61185b69da7573dd4569cc48e1663de75f9f650: Status 404 returned error can't find the container with id b0f2bfa0730f2881d75ac1faf61185b69da7573dd4569cc48e1663de75f9f650 Aug 05 16:28:57.218041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.217954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.218041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.218001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.218268 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.218107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g69vm\" (UniqueName: \"kubernetes.io/projected/4065e8c6-0b56-426b-95b9-65ba00944e43-kube-api-access-g69vm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.319316 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.319277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.319492 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.319327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.319492 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.319377 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69vm\" (UniqueName: \"kubernetes.io/projected/4065e8c6-0b56-426b-95b9-65ba00944e43-kube-api-access-g69vm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.319787 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.319763 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.319850 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.319798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.336885 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.336858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g69vm\" (UniqueName: \"kubernetes.io/projected/4065e8c6-0b56-426b-95b9-65ba00944e43-kube-api-access-g69vm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.373133 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.373079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" event={"ID":"78c88b31-b305-4794-bed6-644030bf41ff","Type":"ContainerStarted","Data":"b0f2bfa0730f2881d75ac1faf61185b69da7573dd4569cc48e1663de75f9f650"} Aug 05 16:28:57.417664 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.417631 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:28:57.572751 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:57.572725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s"] Aug 05 16:28:57.575143 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:28:57.575114 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4065e8c6_0b56_426b_95b9_65ba00944e43.slice/crio-65c6e5a90aea9fe5a1162c66b20f7cf17ebd7a405584a7f67f0dd553419241de WatchSource:0}: Error finding container 65c6e5a90aea9fe5a1162c66b20f7cf17ebd7a405584a7f67f0dd553419241de: Status 404 returned error can't find the container with id 65c6e5a90aea9fe5a1162c66b20f7cf17ebd7a405584a7f67f0dd553419241de Aug 05 16:28:58.378855 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:58.378819 2567 generic.go:356] "Generic (PLEG): container finished" podID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerID="1ca5e9b8636c17b7e36436bbc76bea12758c0af626362459e29220c110b49be7" exitCode=0 Aug 05 16:28:58.379347 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:58.378909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" event={"ID":"4065e8c6-0b56-426b-95b9-65ba00944e43","Type":"ContainerDied","Data":"1ca5e9b8636c17b7e36436bbc76bea12758c0af626362459e29220c110b49be7"} Aug 05 16:28:58.379347 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:28:58.378974 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" event={"ID":"4065e8c6-0b56-426b-95b9-65ba00944e43","Type":"ContainerStarted","Data":"65c6e5a90aea9fe5a1162c66b20f7cf17ebd7a405584a7f67f0dd553419241de"} Aug 05 16:29:00.388981 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:00.388946 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" event={"ID":"78c88b31-b305-4794-bed6-644030bf41ff","Type":"ContainerStarted","Data":"7564f0100d840eaecbe698c201f51f441ce45582339e8501c5eda133ad7c0b40"} Aug 05 16:29:00.389537 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:00.389081 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:29:00.406067 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:00.406016 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" podStartSLOduration=1.6577692640000001 podStartE2EDuration="4.405999752s" totalImagesPullingTime="2.748230488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:00.404540738 +0000 UTC m=+371.864343021" watchObservedRunningTime="2026-08-05 16:29:00.405999752 +0000 UTC m=+371.865802032" Aug 05 16:29:01.394989 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:01.394960 2567 generic.go:356] "Generic (PLEG): container finished" podID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerID="d70e08cea4d2bd3526496ff513ab33ea35b8ad5919b125e53585710c8bd15e35" exitCode=0 Aug 05 16:29:01.395398 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:01.395042 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" event={"ID":"4065e8c6-0b56-426b-95b9-65ba00944e43","Type":"ContainerDied","Data":"d70e08cea4d2bd3526496ff513ab33ea35b8ad5919b125e53585710c8bd15e35"} Aug 05 16:29:02.400560 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:02.400520 2567 generic.go:356] "Generic (PLEG): container finished" podID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerID="cac4f779d64318d3a98743330e9caaff0110b67ac06ef2213554c72781695892" exitCode=0 Aug 05 16:29:02.400987 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:02.400605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" event={"ID":"4065e8c6-0b56-426b-95b9-65ba00944e43","Type":"ContainerDied","Data":"cac4f779d64318d3a98743330e9caaff0110b67ac06ef2213554c72781695892"} Aug 05 16:29:03.525222 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.525198 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:29:03.571473 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.571437 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-util\") pod \"4065e8c6-0b56-426b-95b9-65ba00944e43\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " Aug 05 16:29:03.571623 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.571492 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-bundle\") pod \"4065e8c6-0b56-426b-95b9-65ba00944e43\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " Aug 05 16:29:03.571623 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.571523 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g69vm\" (UniqueName: \"kubernetes.io/projected/4065e8c6-0b56-426b-95b9-65ba00944e43-kube-api-access-g69vm\") pod \"4065e8c6-0b56-426b-95b9-65ba00944e43\" (UID: \"4065e8c6-0b56-426b-95b9-65ba00944e43\") " Aug 05 16:29:03.571910 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.571882 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-bundle" (OuterVolumeSpecName: "bundle") pod "4065e8c6-0b56-426b-95b9-65ba00944e43" (UID: "4065e8c6-0b56-426b-95b9-65ba00944e43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:29:03.573614 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.573583 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4065e8c6-0b56-426b-95b9-65ba00944e43-kube-api-access-g69vm" (OuterVolumeSpecName: "kube-api-access-g69vm") pod "4065e8c6-0b56-426b-95b9-65ba00944e43" (UID: "4065e8c6-0b56-426b-95b9-65ba00944e43"). InnerVolumeSpecName "kube-api-access-g69vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:29:03.578648 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.578622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-util" (OuterVolumeSpecName: "util") pod "4065e8c6-0b56-426b-95b9-65ba00944e43" (UID: "4065e8c6-0b56-426b-95b9-65ba00944e43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:29:03.672634 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.672541 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g69vm\" (UniqueName: \"kubernetes.io/projected/4065e8c6-0b56-426b-95b9-65ba00944e43-kube-api-access-g69vm\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:29:03.672634 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.672572 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:29:03.672634 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:03.672581 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4065e8c6-0b56-426b-95b9-65ba00944e43-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:29:04.410387 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:04.410362 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" Aug 05 16:29:04.410553 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:04.410360 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs8b9s" event={"ID":"4065e8c6-0b56-426b-95b9-65ba00944e43","Type":"ContainerDied","Data":"65c6e5a90aea9fe5a1162c66b20f7cf17ebd7a405584a7f67f0dd553419241de"} Aug 05 16:29:04.410553 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:04.410480 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c6e5a90aea9fe5a1162c66b20f7cf17ebd7a405584a7f67f0dd553419241de" Aug 05 16:29:06.397600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:06.397572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zt86p" Aug 05 16:29:09.718424 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718391 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv"] Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718677 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="extract" Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718691 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="extract" Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718703 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="util" Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718709 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="util" Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718718 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="pull" Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718723 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="pull" Aug 05 16:29:09.718911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.718779 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4065e8c6-0b56-426b-95b9-65ba00944e43" containerName="extract" Aug 05 16:29:09.720584 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.720564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:09.722618 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.722597 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 05 16:29:09.722941 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.722920 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vvmfq\"" Aug 05 16:29:09.723045 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.722955 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:29:09.730410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.730387 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv"] Aug 05 16:29:09.817294 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.817265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zsv\" (UniqueName: \"kubernetes.io/projected/59b28568-c8ba-4bed-8bbc-4abb7e78c433-kube-api-access-n8zsv\") pod \"openshift-lws-operator-fd8ccff4c-q2dtv\" (UID: \"59b28568-c8ba-4bed-8bbc-4abb7e78c433\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:09.817426 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.817333 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b28568-c8ba-4bed-8bbc-4abb7e78c433-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q2dtv\" (UID: \"59b28568-c8ba-4bed-8bbc-4abb7e78c433\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:09.917841 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.917798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b28568-c8ba-4bed-8bbc-4abb7e78c433-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q2dtv\" (UID: \"59b28568-c8ba-4bed-8bbc-4abb7e78c433\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:09.917968 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.917851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zsv\" (UniqueName: \"kubernetes.io/projected/59b28568-c8ba-4bed-8bbc-4abb7e78c433-kube-api-access-n8zsv\") pod \"openshift-lws-operator-fd8ccff4c-q2dtv\" (UID: \"59b28568-c8ba-4bed-8bbc-4abb7e78c433\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:09.918209 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.918189 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b28568-c8ba-4bed-8bbc-4abb7e78c433-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q2dtv\" (UID: \"59b28568-c8ba-4bed-8bbc-4abb7e78c433\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:09.925291 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:09.925267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zsv\" (UniqueName: \"kubernetes.io/projected/59b28568-c8ba-4bed-8bbc-4abb7e78c433-kube-api-access-n8zsv\") pod \"openshift-lws-operator-fd8ccff4c-q2dtv\" (UID: \"59b28568-c8ba-4bed-8bbc-4abb7e78c433\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:10.030769 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:10.030685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" Aug 05 16:29:10.151474 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:10.151381 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv"] Aug 05 16:29:10.154119 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:29:10.154080 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59b28568_c8ba_4bed_8bbc_4abb7e78c433.slice/crio-10debbb6aff87e5d99a4c0cbc5118d4ebc5ebdb4b6761418c1b618d801a7e5ee WatchSource:0}: Error finding container 10debbb6aff87e5d99a4c0cbc5118d4ebc5ebdb4b6761418c1b618d801a7e5ee: Status 404 returned error can't find the container with id 10debbb6aff87e5d99a4c0cbc5118d4ebc5ebdb4b6761418c1b618d801a7e5ee Aug 05 16:29:10.435351 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:10.435316 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" event={"ID":"59b28568-c8ba-4bed-8bbc-4abb7e78c433","Type":"ContainerStarted","Data":"10debbb6aff87e5d99a4c0cbc5118d4ebc5ebdb4b6761418c1b618d801a7e5ee"} Aug 05 16:29:12.444278 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:12.444203 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" event={"ID":"59b28568-c8ba-4bed-8bbc-4abb7e78c433","Type":"ContainerStarted","Data":"0369dca567726ffd010a8ae6ecca143c8caf70a533449b809af5531e7ab9dfa3"} Aug 05 16:29:12.458659 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:12.458601 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2dtv" podStartSLOduration=1.5159491539999999 podStartE2EDuration="3.458581294s" totalImagesPullingTime="1.94263214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:12.457863784 +0000 UTC m=+383.917666065" watchObservedRunningTime="2026-08-05 16:29:12.458581294 +0000 UTC m=+383.918383576" Aug 05 16:29:15.124269 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.124233 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2"] Aug 05 16:29:15.127625 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.127608 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.129755 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.129728 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 05 16:29:15.129755 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.129748 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 05 16:29:15.129911 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.129749 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2cxhs\"" Aug 05 16:29:15.136676 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.136654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2"] Aug 05 16:29:15.255778 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.255747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.255778 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.255785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.255965 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.255869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8l5\" (UniqueName: \"kubernetes.io/projected/3f91c4ec-efd8-493d-b265-6896ac376deb-kube-api-access-lq8l5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.357359 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.357326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.357359 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.357361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.357552 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.357405 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8l5\" (UniqueName: \"kubernetes.io/projected/3f91c4ec-efd8-493d-b265-6896ac376deb-kube-api-access-lq8l5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.357720 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.357700 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.357777 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.357759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.364718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.364693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8l5\" (UniqueName: \"kubernetes.io/projected/3f91c4ec-efd8-493d-b265-6896ac376deb-kube-api-access-lq8l5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.437075 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.436992 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:15.563399 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:15.563325 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2"] Aug 05 16:29:15.565956 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:29:15.565931 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f91c4ec_efd8_493d_b265_6896ac376deb.slice/crio-4f67f01d45551001e0bfba5a82e3f2467d04f0ba61cfa99fa256f080d8122fd6 WatchSource:0}: Error finding container 4f67f01d45551001e0bfba5a82e3f2467d04f0ba61cfa99fa256f080d8122fd6: Status 404 returned error can't find the container with id 4f67f01d45551001e0bfba5a82e3f2467d04f0ba61cfa99fa256f080d8122fd6 Aug 05 16:29:16.460393 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:16.460362 2567 generic.go:356] "Generic (PLEG): container finished" podID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerID="40ac0918c38333c4b0634c349c00d345cd4e10ba61a7032b7abddf09e29b0513" exitCode=0 Aug 05 16:29:16.460799 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:16.460427 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" event={"ID":"3f91c4ec-efd8-493d-b265-6896ac376deb","Type":"ContainerDied","Data":"40ac0918c38333c4b0634c349c00d345cd4e10ba61a7032b7abddf09e29b0513"} Aug 05 16:29:16.460799 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:16.460470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" event={"ID":"3f91c4ec-efd8-493d-b265-6896ac376deb","Type":"ContainerStarted","Data":"4f67f01d45551001e0bfba5a82e3f2467d04f0ba61cfa99fa256f080d8122fd6"} Aug 05 16:29:17.465530 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:17.465448 2567 generic.go:356] "Generic (PLEG): container finished" podID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerID="2869dfd2d3fa60b4218b6f518305441827489695482c902664bcbcaeb27745bf" exitCode=0 Aug 05 16:29:17.465530 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:17.465522 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" event={"ID":"3f91c4ec-efd8-493d-b265-6896ac376deb","Type":"ContainerDied","Data":"2869dfd2d3fa60b4218b6f518305441827489695482c902664bcbcaeb27745bf"} Aug 05 16:29:18.471775 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:18.471740 2567 generic.go:356] "Generic (PLEG): container finished" podID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerID="c1ea1d0358dd41f41996d936a5d56379be0429c2909e31caf578488b6176a8c9" exitCode=0 Aug 05 16:29:18.472195 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:18.471825 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" event={"ID":"3f91c4ec-efd8-493d-b265-6896ac376deb","Type":"ContainerDied","Data":"c1ea1d0358dd41f41996d936a5d56379be0429c2909e31caf578488b6176a8c9"} Aug 05 16:29:19.596801 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.596778 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:19.693433 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.693397 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-util\") pod \"3f91c4ec-efd8-493d-b265-6896ac376deb\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " Aug 05 16:29:19.693572 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.693447 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-bundle\") pod \"3f91c4ec-efd8-493d-b265-6896ac376deb\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " Aug 05 16:29:19.693572 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.693514 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq8l5\" (UniqueName: \"kubernetes.io/projected/3f91c4ec-efd8-493d-b265-6896ac376deb-kube-api-access-lq8l5\") pod \"3f91c4ec-efd8-493d-b265-6896ac376deb\" (UID: \"3f91c4ec-efd8-493d-b265-6896ac376deb\") " Aug 05 16:29:19.694163 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.694134 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-bundle" (OuterVolumeSpecName: "bundle") pod "3f91c4ec-efd8-493d-b265-6896ac376deb" (UID: "3f91c4ec-efd8-493d-b265-6896ac376deb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:29:19.695511 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.695487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f91c4ec-efd8-493d-b265-6896ac376deb-kube-api-access-lq8l5" (OuterVolumeSpecName: "kube-api-access-lq8l5") pod "3f91c4ec-efd8-493d-b265-6896ac376deb" (UID: "3f91c4ec-efd8-493d-b265-6896ac376deb"). InnerVolumeSpecName "kube-api-access-lq8l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:29:19.699134 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.699104 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-util" (OuterVolumeSpecName: "util") pod "3f91c4ec-efd8-493d-b265-6896ac376deb" (UID: "3f91c4ec-efd8-493d-b265-6896ac376deb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:29:19.794301 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.794233 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8l5\" (UniqueName: \"kubernetes.io/projected/3f91c4ec-efd8-493d-b265-6896ac376deb-kube-api-access-lq8l5\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:29:19.794301 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.794259 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:29:19.794301 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:19.794272 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f91c4ec-efd8-493d-b265-6896ac376deb-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:29:20.481386 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:20.481354 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" event={"ID":"3f91c4ec-efd8-493d-b265-6896ac376deb","Type":"ContainerDied","Data":"4f67f01d45551001e0bfba5a82e3f2467d04f0ba61cfa99fa256f080d8122fd6"} Aug 05 16:29:20.481559 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:20.481389 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk7rg2" Aug 05 16:29:20.481559 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:20.481407 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f67f01d45551001e0bfba5a82e3f2467d04f0ba61cfa99fa256f080d8122fd6" Aug 05 16:29:31.595895 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.595860 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf"] Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596159 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="extract" Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596174 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="extract" Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596195 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="util" Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596204 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="util" Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596223 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="pull" Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596229 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="pull" Aug 05 16:29:31.596322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.596270 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f91c4ec-efd8-493d-b265-6896ac376deb" containerName="extract" Aug 05 16:29:31.604505 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.604485 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.606542 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.606522 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 05 16:29:31.606665 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.606555 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 05 16:29:31.606948 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.606930 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sjz7k\"" Aug 05 16:29:31.607728 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.607711 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 05 16:29:31.608910 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.608896 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 05 16:29:31.613941 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.613917 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf"] Aug 05 16:29:31.690253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.690220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c0047bc-7177-4240-881e-a11afe1b29ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.690407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.690267 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c0047bc-7177-4240-881e-a11afe1b29ba-manifests\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.690407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.690294 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c0047bc-7177-4240-881e-a11afe1b29ba-tmp\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.690407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.690319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7f5c\" (UniqueName: \"kubernetes.io/projected/0c0047bc-7177-4240-881e-a11afe1b29ba-kube-api-access-q7f5c\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.690407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.690358 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c0047bc-7177-4240-881e-a11afe1b29ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.791360 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.791330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c0047bc-7177-4240-881e-a11afe1b29ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.791555 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.791383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c0047bc-7177-4240-881e-a11afe1b29ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.791555 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.791413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c0047bc-7177-4240-881e-a11afe1b29ba-manifests\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.791555 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.791431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c0047bc-7177-4240-881e-a11afe1b29ba-tmp\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.791555 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.791449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7f5c\" (UniqueName: \"kubernetes.io/projected/0c0047bc-7177-4240-881e-a11afe1b29ba-kube-api-access-q7f5c\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.791899 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.791878 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c0047bc-7177-4240-881e-a11afe1b29ba-manifests\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.793651 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.793632 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c0047bc-7177-4240-881e-a11afe1b29ba-tmp\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.793715 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.793699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c0047bc-7177-4240-881e-a11afe1b29ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.793952 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.793933 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c0047bc-7177-4240-881e-a11afe1b29ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.802345 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.802319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7f5c\" (UniqueName: \"kubernetes.io/projected/0c0047bc-7177-4240-881e-a11afe1b29ba-kube-api-access-q7f5c\") pod \"opendatahub-operator-controller-manager-5cfc779b6-fmstf\" (UID: \"0c0047bc-7177-4240-881e-a11afe1b29ba\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:31.914173 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:31.914112 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:32.046743 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:32.046723 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf"] Aug 05 16:29:32.049358 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:29:32.049335 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0047bc_7177_4240_881e_a11afe1b29ba.slice/crio-1d7fde8d39606ea3732e787462394de130c27de3e6b8742b492a5fdfce28eeac WatchSource:0}: Error finding container 1d7fde8d39606ea3732e787462394de130c27de3e6b8742b492a5fdfce28eeac: Status 404 returned error can't find the container with id 1d7fde8d39606ea3732e787462394de130c27de3e6b8742b492a5fdfce28eeac Aug 05 16:29:32.535519 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:32.535474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" event={"ID":"0c0047bc-7177-4240-881e-a11afe1b29ba","Type":"ContainerStarted","Data":"1d7fde8d39606ea3732e787462394de130c27de3e6b8742b492a5fdfce28eeac"} Aug 05 16:29:35.190441 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:35.190400 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6595c5965c-xpbgf"] Aug 05 16:29:35.553823 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:35.553743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" event={"ID":"0c0047bc-7177-4240-881e-a11afe1b29ba","Type":"ContainerStarted","Data":"13a53860bf834fe9aa31f93127b63761ea1ca3feb9499fc91850357553080a3f"} Aug 05 16:29:36.558759 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:36.558725 2567 generic.go:356] "Generic (PLEG): container finished" podID="0c0047bc-7177-4240-881e-a11afe1b29ba" containerID="13a53860bf834fe9aa31f93127b63761ea1ca3feb9499fc91850357553080a3f" exitCode=0 Aug 05 16:29:36.559151 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:36.558776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" event={"ID":"0c0047bc-7177-4240-881e-a11afe1b29ba","Type":"ContainerDied","Data":"13a53860bf834fe9aa31f93127b63761ea1ca3feb9499fc91850357553080a3f"} Aug 05 16:29:37.565317 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:37.565278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" event={"ID":"0c0047bc-7177-4240-881e-a11afe1b29ba","Type":"ContainerStarted","Data":"8730538f6f25a36edcdb4ee864b32d5bce15f53967ffa416ad13de631371a7fb"} Aug 05 16:29:37.565715 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:37.565405 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:37.586154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:37.586080 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" podStartSLOduration=2.369078767 podStartE2EDuration="6.58606633s" totalImagesPullingTime="3.121352647s" totalInitContainerRuntime="1.095634916s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:37.583864579 +0000 UTC m=+409.043666860" watchObservedRunningTime="2026-08-05 16:29:37.58606633 +0000 UTC m=+409.045868682" Aug 05 16:29:48.571970 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:48.571936 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cfc779b6-fmstf" Aug 05 16:29:51.182023 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.181990 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4zqsq"] Aug 05 16:29:51.187437 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.187417 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.192569 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.192545 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 05 16:29:51.192678 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.192570 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dm65k\"" Aug 05 16:29:51.198123 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.198101 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4zqsq"] Aug 05 16:29:51.236842 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.236817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd764260-49cf-46a1-885e-a3970fc15a17-cert\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.236965 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.236857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgp8\" (UniqueName: \"kubernetes.io/projected/fd764260-49cf-46a1-885e-a3970fc15a17-kube-api-access-5pgp8\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.337661 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.337632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd764260-49cf-46a1-885e-a3970fc15a17-cert\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.337837 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.337675 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgp8\" (UniqueName: \"kubernetes.io/projected/fd764260-49cf-46a1-885e-a3970fc15a17-kube-api-access-5pgp8\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.337837 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:29:51.337789 2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 05 16:29:51.337964 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:29:51.337892 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd764260-49cf-46a1-885e-a3970fc15a17-cert podName:fd764260-49cf-46a1-885e-a3970fc15a17 nodeName:}" failed. No retries permitted until 2026-08-05 16:29:51.8378687 +0000 UTC m=+423.297670966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fd764260-49cf-46a1-885e-a3970fc15a17-cert") pod "odh-model-controller-7998c67dc4-4zqsq" (UID: "fd764260-49cf-46a1-885e-a3970fc15a17") : secret "odh-model-controller-webhook-cert" not found Aug 05 16:29:51.345982 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.345958 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgp8\" (UniqueName: \"kubernetes.io/projected/fd764260-49cf-46a1-885e-a3970fc15a17-kube-api-access-5pgp8\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.840174 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.840124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd764260-49cf-46a1-885e-a3970fc15a17-cert\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.842469 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.842445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd764260-49cf-46a1-885e-a3970fc15a17-cert\") pod \"odh-model-controller-7998c67dc4-4zqsq\" (UID: \"fd764260-49cf-46a1-885e-a3970fc15a17\") " pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:51.937075 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.937044 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4c85t"] Aug 05 16:29:51.940558 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.940529 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:51.943110 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.943073 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 05 16:29:51.954595 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:51.954570 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4c85t"] Aug 05 16:29:52.040999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.040959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e0b62e8-763c-40e0-b44b-20ba4aefdbad-tls-certs\") pod \"model-serving-api-54d89bd5fc-4c85t\" (UID: \"3e0b62e8-763c-40e0-b44b-20ba4aefdbad\") " pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.041217 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.041019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ddx\" (UniqueName: \"kubernetes.io/projected/3e0b62e8-763c-40e0-b44b-20ba4aefdbad-kube-api-access-j6ddx\") pod \"model-serving-api-54d89bd5fc-4c85t\" (UID: \"3e0b62e8-763c-40e0-b44b-20ba4aefdbad\") " pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.098167 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.098070 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:52.141437 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.141401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e0b62e8-763c-40e0-b44b-20ba4aefdbad-tls-certs\") pod \"model-serving-api-54d89bd5fc-4c85t\" (UID: \"3e0b62e8-763c-40e0-b44b-20ba4aefdbad\") " pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.141618 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.141507 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ddx\" (UniqueName: \"kubernetes.io/projected/3e0b62e8-763c-40e0-b44b-20ba4aefdbad-kube-api-access-j6ddx\") pod \"model-serving-api-54d89bd5fc-4c85t\" (UID: \"3e0b62e8-763c-40e0-b44b-20ba4aefdbad\") " pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.144445 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.144401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e0b62e8-763c-40e0-b44b-20ba4aefdbad-tls-certs\") pod \"model-serving-api-54d89bd5fc-4c85t\" (UID: \"3e0b62e8-763c-40e0-b44b-20ba4aefdbad\") " pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.150249 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.150219 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ddx\" (UniqueName: \"kubernetes.io/projected/3e0b62e8-763c-40e0-b44b-20ba4aefdbad-kube-api-access-j6ddx\") pod \"model-serving-api-54d89bd5fc-4c85t\" (UID: \"3e0b62e8-763c-40e0-b44b-20ba4aefdbad\") " pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.233985 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.233960 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4zqsq"] Aug 05 16:29:52.236839 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:29:52.236809 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd764260_49cf_46a1_885e_a3970fc15a17.slice/crio-15d3e5edc633cf6015fbc8236197bf0bd90460c9a151e9fc8878bc47280e93a9 WatchSource:0}: Error finding container 15d3e5edc633cf6015fbc8236197bf0bd90460c9a151e9fc8878bc47280e93a9: Status 404 returned error can't find the container with id 15d3e5edc633cf6015fbc8236197bf0bd90460c9a151e9fc8878bc47280e93a9 Aug 05 16:29:52.254064 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.254044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:52.389986 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.389931 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4c85t"] Aug 05 16:29:52.392401 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:29:52.392371 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0b62e8_763c_40e0_b44b_20ba4aefdbad.slice/crio-c82f7bbb951e5a724ebb68c9a2d24002bb377fbbd7a79271bfeae9473e19c127 WatchSource:0}: Error finding container c82f7bbb951e5a724ebb68c9a2d24002bb377fbbd7a79271bfeae9473e19c127: Status 404 returned error can't find the container with id c82f7bbb951e5a724ebb68c9a2d24002bb377fbbd7a79271bfeae9473e19c127 Aug 05 16:29:52.632779 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.632742 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" event={"ID":"3e0b62e8-763c-40e0-b44b-20ba4aefdbad","Type":"ContainerStarted","Data":"c82f7bbb951e5a724ebb68c9a2d24002bb377fbbd7a79271bfeae9473e19c127"} Aug 05 16:29:52.633996 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:52.633971 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" event={"ID":"fd764260-49cf-46a1-885e-a3970fc15a17","Type":"ContainerStarted","Data":"15d3e5edc633cf6015fbc8236197bf0bd90460c9a151e9fc8878bc47280e93a9"} Aug 05 16:29:56.657852 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:56.657782 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" event={"ID":"fd764260-49cf-46a1-885e-a3970fc15a17","Type":"ContainerStarted","Data":"ec4bd82c919cd7c0db8f575bc64ceb908a9c0ea10f7cfecf096623bbabdc086e"} Aug 05 16:29:56.658395 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:56.657971 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:29:56.659824 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:56.659801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" event={"ID":"3e0b62e8-763c-40e0-b44b-20ba4aefdbad","Type":"ContainerStarted","Data":"e2e3e237060853b265ba814e4b010ff2996127675a31031273459d17a10ae789"} Aug 05 16:29:56.659953 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:56.659939 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:29:56.674943 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:56.674905 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" podStartSLOduration=1.708019988 podStartE2EDuration="5.674891496s" totalImagesPullingTime="3.966871508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:56.673212783 +0000 UTC m=+428.133015077" watchObservedRunningTime="2026-08-05 16:29:56.674891496 +0000 UTC m=+428.134693801" Aug 05 16:29:56.688398 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:29:56.688358 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" podStartSLOduration=1.833471407 podStartE2EDuration="5.688344669s" totalImagesPullingTime="3.854873262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:29:56.687175451 +0000 UTC m=+428.146977756" watchObservedRunningTime="2026-08-05 16:29:56.688344669 +0000 UTC m=+428.148146950" Aug 05 16:30:00.030646 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.030604 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk"] Aug 05 16:30:00.034051 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.034030 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.037042 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.037023 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 05 16:30:00.037332 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.037314 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 05 16:30:00.039142 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.039125 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 05 16:30:00.039209 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.039146 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 05 16:30:00.039671 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.039651 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 05 16:30:00.039770 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.039683 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gjzpv\"" Aug 05 16:30:00.039770 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.039654 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 05 16:30:00.048855 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.048829 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk"] Aug 05 16:30:00.051957 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.051932 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw"] Aug 05 16:30:00.055362 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.055345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.057438 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.057419 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 05 16:30:00.057550 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.057456 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-49q4h\"" Aug 05 16:30:00.057550 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.057468 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 05 16:30:00.057550 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.057458 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 05 16:30:00.068609 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.068587 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw"] Aug 05 16:30:00.113543 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113515 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.113697 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.113697 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/168a0b8c-52d5-4f2c-9502-eed982eb37a3-metrics-cert\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.113697 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.113697 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/168a0b8c-52d5-4f2c-9502-eed982eb37a3-cert\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.113697 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113681 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864279c8-eb32-4d0b-b3c7-f1690e160f3f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.113908 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkqq\" (UniqueName: \"kubernetes.io/projected/168a0b8c-52d5-4f2c-9502-eed982eb37a3-kube-api-access-swkqq\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.113908 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.113908 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.113908 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/168a0b8c-52d5-4f2c-9502-eed982eb37a3-manager-config\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.113908 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.113830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqmd\" (UniqueName: \"kubernetes.io/projected/864279c8-eb32-4d0b-b3c7-f1690e160f3f-kube-api-access-4fqmd\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.209000 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.208954 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6595c5965c-xpbgf" podUID="817874d4-92e5-4c57-80a0-1e8ec74b39c7" containerName="console" containerID="cri-o://213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5" gracePeriod=15 Aug 05 16:30:00.214853 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.214828 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.214953 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.214877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/168a0b8c-52d5-4f2c-9502-eed982eb37a3-cert\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.215061 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215037 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864279c8-eb32-4d0b-b3c7-f1690e160f3f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.215142 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkqq\" (UniqueName: \"kubernetes.io/projected/168a0b8c-52d5-4f2c-9502-eed982eb37a3-kube-api-access-swkqq\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.215197 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.215253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.215253 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/168a0b8c-52d5-4f2c-9502-eed982eb37a3-manager-config\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.215349 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215254 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqmd\" (UniqueName: \"kubernetes.io/projected/864279c8-eb32-4d0b-b3c7-f1690e160f3f-kube-api-access-4fqmd\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.215349 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.215439 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.215439 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.215397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/168a0b8c-52d5-4f2c-9502-eed982eb37a3-metrics-cert\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.216552 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.216525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.216694 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.216553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/168a0b8c-52d5-4f2c-9502-eed982eb37a3-manager-config\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.217897 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.217872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.218248 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.218225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/864279c8-eb32-4d0b-b3c7-f1690e160f3f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.218365 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.218280 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.218365 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.218350 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.218479 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.218465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/168a0b8c-52d5-4f2c-9502-eed982eb37a3-cert\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.218916 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.218873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/168a0b8c-52d5-4f2c-9502-eed982eb37a3-metrics-cert\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.228293 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.228275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkqq\" (UniqueName: \"kubernetes.io/projected/168a0b8c-52d5-4f2c-9502-eed982eb37a3-kube-api-access-swkqq\") pod \"lws-controller-manager-6bfcf8b475-kxtlw\" (UID: \"168a0b8c-52d5-4f2c-9502-eed982eb37a3\") " pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.229918 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.229902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/864279c8-eb32-4d0b-b3c7-f1690e160f3f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.236144 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.236126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqmd\" (UniqueName: \"kubernetes.io/projected/864279c8-eb32-4d0b-b3c7-f1690e160f3f-kube-api-access-4fqmd\") pod \"istiod-openshift-gateway-5677ccb49b-xl9xk\" (UID: \"864279c8-eb32-4d0b-b3c7-f1690e160f3f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.344713 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.344689 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:00.364759 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.364737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:00.468112 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.468070 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:30:00.492327 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.492296 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk"] Aug 05 16:30:00.497114 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:30:00.497074 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864279c8_eb32_4d0b_b3c7_f1690e160f3f.slice/crio-e76a7a086b39cdfe6e5dbe94f76c29fa150b1d4b7b06788b008fe2a82f5e4fdf WatchSource:0}: Error finding container e76a7a086b39cdfe6e5dbe94f76c29fa150b1d4b7b06788b008fe2a82f5e4fdf: Status 404 returned error can't find the container with id e76a7a086b39cdfe6e5dbe94f76c29fa150b1d4b7b06788b008fe2a82f5e4fdf Aug 05 16:30:00.517284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517259 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-serving-cert\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.517410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517290 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-oauth-config\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.517410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517321 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-config\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.517410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517350 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l4pr\" (UniqueName: \"kubernetes.io/projected/817874d4-92e5-4c57-80a0-1e8ec74b39c7-kube-api-access-9l4pr\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.517410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517373 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-oauth-serving-cert\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.517683 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517434 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-trusted-ca-bundle\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.517683 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517478 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-service-ca\") pod \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\" (UID: \"817874d4-92e5-4c57-80a0-1e8ec74b39c7\") " Aug 05 16:30:00.518067 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.517950 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:30:00.518067 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.518049 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:30:00.518250 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.518223 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-config" (OuterVolumeSpecName: "console-config") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:30:00.518437 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.518397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:30:00.518676 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.518656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw"] Aug 05 16:30:00.519623 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.519600 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:30:00.519849 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.519833 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:30:00.520114 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.520081 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817874d4-92e5-4c57-80a0-1e8ec74b39c7-kube-api-access-9l4pr" (OuterVolumeSpecName: "kube-api-access-9l4pr") pod "817874d4-92e5-4c57-80a0-1e8ec74b39c7" (UID: "817874d4-92e5-4c57-80a0-1e8ec74b39c7"). InnerVolumeSpecName "kube-api-access-9l4pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:00.520726 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:30:00.520705 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168a0b8c_52d5_4f2c_9502_eed982eb37a3.slice/crio-90ef3a4fa24eee21417a105312bea036e5702060d9d543fd98823f37d45c9171 WatchSource:0}: Error finding container 90ef3a4fa24eee21417a105312bea036e5702060d9d543fd98823f37d45c9171: Status 404 returned error can't find the container with id 90ef3a4fa24eee21417a105312bea036e5702060d9d543fd98823f37d45c9171 Aug 05 16:30:00.618270 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618246 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-trusted-ca-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.618270 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618270 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.618462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618281 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-serving-cert\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.618462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618292 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-oauth-config\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.618462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618302 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-console-config\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.618462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618311 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4pr\" (UniqueName: \"kubernetes.io/projected/817874d4-92e5-4c57-80a0-1e8ec74b39c7-kube-api-access-9l4pr\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.618462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.618319 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817874d4-92e5-4c57-80a0-1e8ec74b39c7-oauth-serving-cert\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:00.679040 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.679008 2567 generic.go:356] "Generic (PLEG): container finished" podID="817874d4-92e5-4c57-80a0-1e8ec74b39c7" containerID="213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5" exitCode=0 Aug 05 16:30:00.679214 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.679060 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6595c5965c-xpbgf" Aug 05 16:30:00.679214 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.679110 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6595c5965c-xpbgf" event={"ID":"817874d4-92e5-4c57-80a0-1e8ec74b39c7","Type":"ContainerDied","Data":"213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5"} Aug 05 16:30:00.679214 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.679165 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6595c5965c-xpbgf" event={"ID":"817874d4-92e5-4c57-80a0-1e8ec74b39c7","Type":"ContainerDied","Data":"f17aa8163aa80eeaa51d7030bb7f51a4e2b2ac6733035c5e540b9628725584b7"} Aug 05 16:30:00.679214 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.679192 2567 scope.go:117] "RemoveContainer" containerID="213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5" Aug 05 16:30:00.680514 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.680490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" event={"ID":"168a0b8c-52d5-4f2c-9502-eed982eb37a3","Type":"ContainerStarted","Data":"90ef3a4fa24eee21417a105312bea036e5702060d9d543fd98823f37d45c9171"} Aug 05 16:30:00.681799 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.681778 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" event={"ID":"864279c8-eb32-4d0b-b3c7-f1690e160f3f","Type":"ContainerStarted","Data":"e76a7a086b39cdfe6e5dbe94f76c29fa150b1d4b7b06788b008fe2a82f5e4fdf"} Aug 05 16:30:00.688138 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.688121 2567 scope.go:117] "RemoveContainer" containerID="213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5" Aug 05 16:30:00.688392 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:30:00.688377 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5\": container with ID starting with 213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5 not found: ID does not exist" containerID="213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5" Aug 05 16:30:00.688440 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.688398 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5"} err="failed to get container status \"213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5\": rpc error: code = NotFound desc = could not find container \"213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5\": container with ID starting with 213bb125e4c70c6309c0be1a257fbfd3060f0515a393b27cf34af825aa89bcd5 not found: ID does not exist" Aug 05 16:30:00.713310 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.713288 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6595c5965c-xpbgf"] Aug 05 16:30:00.717479 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:00.717459 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6595c5965c-xpbgf"] Aug 05 16:30:01.011119 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:01.011022 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817874d4-92e5-4c57-80a0-1e8ec74b39c7" path="/var/lib/kubelet/pods/817874d4-92e5-4c57-80a0-1e8ec74b39c7/volumes" Aug 05 16:30:03.645908 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:03.645860 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Aug 05 16:30:03.646292 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:03.645965 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Aug 05 16:30:04.708296 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.708261 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" event={"ID":"168a0b8c-52d5-4f2c-9502-eed982eb37a3","Type":"ContainerStarted","Data":"7c77b575d3d4da9b7a7c6ef7e26d7aaa37c9988a46d83f378fcd1cc58b2c0584"} Aug 05 16:30:04.708725 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.708318 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:04.709923 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.709901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" event={"ID":"864279c8-eb32-4d0b-b3c7-f1690e160f3f","Type":"ContainerStarted","Data":"cdaad57d6d4a6bbc538dae7a9f53522b85ddf09a8e52818d188fd81fa4c90e55"} Aug 05 16:30:04.710128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.710111 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:04.711862 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.711836 2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-xl9xk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 16:30:04.711957 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.711884 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" podUID="864279c8-eb32-4d0b-b3c7-f1690e160f3f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 16:30:04.725552 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.725499 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" podStartSLOduration=1.607369411 podStartE2EDuration="4.725485715s" totalImagesPullingTime="3.118116304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:04.723721513 +0000 UTC m=+436.183523794" watchObservedRunningTime="2026-08-05 16:30:04.725485715 +0000 UTC m=+436.185287995" Aug 05 16:30:04.745165 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:04.745119 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" podStartSLOduration=1.598745542 podStartE2EDuration="4.745105758s" totalImagesPullingTime="3.146360216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:04.743526469 +0000 UTC m=+436.203328752" watchObservedRunningTime="2026-08-05 16:30:04.745105758 +0000 UTC m=+436.204908030" Aug 05 16:30:05.715987 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:05.715952 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xl9xk" Aug 05 16:30:07.667781 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:07.667753 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4zqsq" Aug 05 16:30:07.669745 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:07.669725 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4c85t" Aug 05 16:30:15.717633 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:15.717552 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bfcf8b475-kxtlw" Aug 05 16:30:55.334777 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.334730 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65"] Aug 05 16:30:55.335305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.335037 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817874d4-92e5-4c57-80a0-1e8ec74b39c7" containerName="console" Aug 05 16:30:55.335305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.335050 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="817874d4-92e5-4c57-80a0-1e8ec74b39c7" containerName="console" Aug 05 16:30:55.335305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.335122 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="817874d4-92e5-4c57-80a0-1e8ec74b39c7" containerName="console" Aug 05 16:30:55.337704 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.337687 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.339552 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.339532 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 05 16:30:55.339636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.339532 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2cxhs\"" Aug 05 16:30:55.339828 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.339809 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 05 16:30:55.344219 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.344191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65"] Aug 05 16:30:55.422553 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.422521 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.422726 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.422558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.422726 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.422605 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwb2\" (UniqueName: \"kubernetes.io/projected/74e2a254-0bdd-452e-987b-6d2f6502e82a-kube-api-access-7gwb2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.431706 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.431671 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg"] Aug 05 16:30:55.434216 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.434199 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.443385 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.443359 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg"] Aug 05 16:30:55.523358 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.523561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.523561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523390 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.523561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxms\" (UniqueName: \"kubernetes.io/projected/47e8d448-9e64-4c06-b945-b16289a844ae-kube-api-access-6vxms\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.523561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523442 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.523561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwb2\" (UniqueName: \"kubernetes.io/projected/74e2a254-0bdd-452e-987b-6d2f6502e82a-kube-api-access-7gwb2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.523830 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.523830 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.523804 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.532836 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.532797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwb2\" (UniqueName: \"kubernetes.io/projected/74e2a254-0bdd-452e-987b-6d2f6502e82a-kube-api-access-7gwb2\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.536471 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.536442 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl"] Aug 05 16:30:55.538862 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.538840 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.546600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.546573 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl"] Aug 05 16:30:55.624191 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624116 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxms\" (UniqueName: \"kubernetes.io/projected/47e8d448-9e64-4c06-b945-b16289a844ae-kube-api-access-6vxms\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.624191 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwdb\" (UniqueName: \"kubernetes.io/projected/c43db9f0-9339-4f82-9a82-d98da49f5d48-kube-api-access-gpwdb\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.624191 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.624394 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624232 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.624394 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.624394 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.624514 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.624646 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.624631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.632656 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.632629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxms\" (UniqueName: \"kubernetes.io/projected/47e8d448-9e64-4c06-b945-b16289a844ae-kube-api-access-6vxms\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.632846 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.632828 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb"] Aug 05 16:30:55.635138 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.635121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.642587 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.642560 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb"] Aug 05 16:30:55.650015 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.649962 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:55.726306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.725826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwdb\" (UniqueName: \"kubernetes.io/projected/c43db9f0-9339-4f82-9a82-d98da49f5d48-kube-api-access-gpwdb\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.726306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.725920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.726306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.725977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.726306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.726063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.726306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.726136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.726306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.726166 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lv4m\" (UniqueName: \"kubernetes.io/projected/29587b65-1f3d-4e28-8545-1bb73a761555-kube-api-access-5lv4m\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.726662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.726402 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.726662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.726597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.736306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.736277 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwdb\" (UniqueName: \"kubernetes.io/projected/c43db9f0-9339-4f82-9a82-d98da49f5d48-kube-api-access-gpwdb\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.744026 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.744003 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:30:55.799809 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.799758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65"] Aug 05 16:30:55.827477 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.827456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.827538 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.827497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.827538 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.827528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lv4m\" (UniqueName: \"kubernetes.io/projected/29587b65-1f3d-4e28-8545-1bb73a761555-kube-api-access-5lv4m\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.827877 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.827856 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.827956 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.827882 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.838075 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.838046 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lv4m\" (UniqueName: \"kubernetes.io/projected/29587b65-1f3d-4e28-8545-1bb73a761555-kube-api-access-5lv4m\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.848910 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.848598 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:30:55.883470 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.883335 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg"] Aug 05 16:30:55.886581 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:30:55.886551 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e8d448_9e64_4c06_b945_b16289a844ae.slice/crio-24cf923977694c9600f0fc8b8e18366db1232c3342bac2b7323dabbb79361ec8 WatchSource:0}: Error finding container 24cf923977694c9600f0fc8b8e18366db1232c3342bac2b7323dabbb79361ec8: Status 404 returned error can't find the container with id 24cf923977694c9600f0fc8b8e18366db1232c3342bac2b7323dabbb79361ec8 Aug 05 16:30:55.936385 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.936350 2567 generic.go:356] "Generic (PLEG): container finished" podID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerID="e0407eed4d59aafc422cb7f585f72bdfc10af824b1223bb3bcf1ee665318fc37" exitCode=0 Aug 05 16:30:55.936551 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.936398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" event={"ID":"74e2a254-0bdd-452e-987b-6d2f6502e82a","Type":"ContainerDied","Data":"e0407eed4d59aafc422cb7f585f72bdfc10af824b1223bb3bcf1ee665318fc37"} Aug 05 16:30:55.936551 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.936448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" event={"ID":"74e2a254-0bdd-452e-987b-6d2f6502e82a","Type":"ContainerStarted","Data":"d618e5762b812b24affba20e66e6dc051cecc7966dd2175a7048557973d016e2"} Aug 05 16:30:55.938218 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.938174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" event={"ID":"47e8d448-9e64-4c06-b945-b16289a844ae","Type":"ContainerStarted","Data":"24cf923977694c9600f0fc8b8e18366db1232c3342bac2b7323dabbb79361ec8"} Aug 05 16:30:55.965199 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.965170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:30:55.996707 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:55.996677 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl"] Aug 05 16:30:55.999326 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:30:55.999283 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43db9f0_9339_4f82_9a82_d98da49f5d48.slice/crio-b1e5b5dc31746afafc9e6106e66230a680b25833bc176a287abce389e493ad90 WatchSource:0}: Error finding container b1e5b5dc31746afafc9e6106e66230a680b25833bc176a287abce389e493ad90: Status 404 returned error can't find the container with id b1e5b5dc31746afafc9e6106e66230a680b25833bc176a287abce389e493ad90 Aug 05 16:30:56.113930 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.113904 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb"] Aug 05 16:30:56.172144 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:30:56.172051 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29587b65_1f3d_4e28_8545_1bb73a761555.slice/crio-923e29b3c9c105d218134ccaa946c3411ef4ddbb86300d003526f6690510d6af WatchSource:0}: Error finding container 923e29b3c9c105d218134ccaa946c3411ef4ddbb86300d003526f6690510d6af: Status 404 returned error can't find the container with id 923e29b3c9c105d218134ccaa946c3411ef4ddbb86300d003526f6690510d6af Aug 05 16:30:56.944458 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.944424 2567 generic.go:356] "Generic (PLEG): container finished" podID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerID="c27b662108e7ad19fe2b58e50be15fba1ed4dadd53d11ea0a814802b37ea3416" exitCode=0 Aug 05 16:30:56.944916 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.944510 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" event={"ID":"74e2a254-0bdd-452e-987b-6d2f6502e82a","Type":"ContainerDied","Data":"c27b662108e7ad19fe2b58e50be15fba1ed4dadd53d11ea0a814802b37ea3416"} Aug 05 16:30:56.946104 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.946070 2567 generic.go:356] "Generic (PLEG): container finished" podID="47e8d448-9e64-4c06-b945-b16289a844ae" containerID="fd1a1b47aea9171d4c853499c050eab73e909e75afb4af664cacdb8c6f0dedf9" exitCode=0 Aug 05 16:30:56.946187 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.946116 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" event={"ID":"47e8d448-9e64-4c06-b945-b16289a844ae","Type":"ContainerDied","Data":"fd1a1b47aea9171d4c853499c050eab73e909e75afb4af664cacdb8c6f0dedf9"} Aug 05 16:30:56.947625 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.947607 2567 generic.go:356] "Generic (PLEG): container finished" podID="29587b65-1f3d-4e28-8545-1bb73a761555" containerID="7af68b3aea21ac451dd685c7d8543b9723d6a105fd0cbb7f4263a0d7dc2efcb4" exitCode=0 Aug 05 16:30:56.947691 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.947671 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" event={"ID":"29587b65-1f3d-4e28-8545-1bb73a761555","Type":"ContainerDied","Data":"7af68b3aea21ac451dd685c7d8543b9723d6a105fd0cbb7f4263a0d7dc2efcb4"} Aug 05 16:30:56.947746 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.947705 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" event={"ID":"29587b65-1f3d-4e28-8545-1bb73a761555","Type":"ContainerStarted","Data":"923e29b3c9c105d218134ccaa946c3411ef4ddbb86300d003526f6690510d6af"} Aug 05 16:30:56.949386 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.949341 2567 generic.go:356] "Generic (PLEG): container finished" podID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerID="fe569aeaf8dfaa4aaedc8df9f975b6af6f0cb7e3a0b26e6b1945088710a45bca" exitCode=0 Aug 05 16:30:56.949386 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.949367 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" event={"ID":"c43db9f0-9339-4f82-9a82-d98da49f5d48","Type":"ContainerDied","Data":"fe569aeaf8dfaa4aaedc8df9f975b6af6f0cb7e3a0b26e6b1945088710a45bca"} Aug 05 16:30:56.949493 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:56.949400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" event={"ID":"c43db9f0-9339-4f82-9a82-d98da49f5d48","Type":"ContainerStarted","Data":"b1e5b5dc31746afafc9e6106e66230a680b25833bc176a287abce389e493ad90"} Aug 05 16:30:57.958793 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:57.958763 2567 generic.go:356] "Generic (PLEG): container finished" podID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerID="2cb18bd5695822f52264b85a71364930e1bae74e21b31eb118616ff812c2f8fc" exitCode=0 Aug 05 16:30:57.959246 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:57.958849 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" event={"ID":"74e2a254-0bdd-452e-987b-6d2f6502e82a","Type":"ContainerDied","Data":"2cb18bd5695822f52264b85a71364930e1bae74e21b31eb118616ff812c2f8fc"} Aug 05 16:30:57.960866 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:57.960843 2567 generic.go:356] "Generic (PLEG): container finished" podID="47e8d448-9e64-4c06-b945-b16289a844ae" containerID="d57b014354e277a778ed07b36169ac43462116c2a9af12ce76ea5718bc85138b" exitCode=0 Aug 05 16:30:57.960981 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:57.960876 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" event={"ID":"47e8d448-9e64-4c06-b945-b16289a844ae","Type":"ContainerDied","Data":"d57b014354e277a778ed07b36169ac43462116c2a9af12ce76ea5718bc85138b"} Aug 05 16:30:58.967157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:58.967054 2567 generic.go:356] "Generic (PLEG): container finished" podID="47e8d448-9e64-4c06-b945-b16289a844ae" containerID="2f177bade01d1639e3829864ed72f3b6d9d159a68d74d75144a52de229c122db" exitCode=0 Aug 05 16:30:58.967157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:58.967144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" event={"ID":"47e8d448-9e64-4c06-b945-b16289a844ae","Type":"ContainerDied","Data":"2f177bade01d1639e3829864ed72f3b6d9d159a68d74d75144a52de229c122db"} Aug 05 16:30:58.968887 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:58.968863 2567 generic.go:356] "Generic (PLEG): container finished" podID="29587b65-1f3d-4e28-8545-1bb73a761555" containerID="be8eeab276dd3938cc59a0df8a6bba58815c0bfb77c0e453000e1049d768d290" exitCode=0 Aug 05 16:30:58.968983 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:58.968948 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" event={"ID":"29587b65-1f3d-4e28-8545-1bb73a761555","Type":"ContainerDied","Data":"be8eeab276dd3938cc59a0df8a6bba58815c0bfb77c0e453000e1049d768d290"} Aug 05 16:30:58.970688 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:58.970671 2567 generic.go:356] "Generic (PLEG): container finished" podID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerID="4384afdbf6c9344f171f8fc72f8c9c50e7082feaf46cff0f10f5f2deeb5e12bd" exitCode=0 Aug 05 16:30:58.970764 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:58.970730 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" event={"ID":"c43db9f0-9339-4f82-9a82-d98da49f5d48","Type":"ContainerDied","Data":"4384afdbf6c9344f171f8fc72f8c9c50e7082feaf46cff0f10f5f2deeb5e12bd"} Aug 05 16:30:59.114241 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.114217 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:59.154936 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.154908 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-util\") pod \"74e2a254-0bdd-452e-987b-6d2f6502e82a\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " Aug 05 16:30:59.155041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.154993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-bundle\") pod \"74e2a254-0bdd-452e-987b-6d2f6502e82a\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " Aug 05 16:30:59.155041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.155033 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gwb2\" (UniqueName: \"kubernetes.io/projected/74e2a254-0bdd-452e-987b-6d2f6502e82a-kube-api-access-7gwb2\") pod \"74e2a254-0bdd-452e-987b-6d2f6502e82a\" (UID: \"74e2a254-0bdd-452e-987b-6d2f6502e82a\") " Aug 05 16:30:59.155505 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.155478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-bundle" (OuterVolumeSpecName: "bundle") pod "74e2a254-0bdd-452e-987b-6d2f6502e82a" (UID: "74e2a254-0bdd-452e-987b-6d2f6502e82a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:59.156982 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.156963 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e2a254-0bdd-452e-987b-6d2f6502e82a-kube-api-access-7gwb2" (OuterVolumeSpecName: "kube-api-access-7gwb2") pod "74e2a254-0bdd-452e-987b-6d2f6502e82a" (UID: "74e2a254-0bdd-452e-987b-6d2f6502e82a"). InnerVolumeSpecName "kube-api-access-7gwb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:59.161491 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.161455 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-util" (OuterVolumeSpecName: "util") pod "74e2a254-0bdd-452e-987b-6d2f6502e82a" (UID: "74e2a254-0bdd-452e-987b-6d2f6502e82a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:59.255845 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.255763 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:59.255845 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.255800 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gwb2\" (UniqueName: \"kubernetes.io/projected/74e2a254-0bdd-452e-987b-6d2f6502e82a-kube-api-access-7gwb2\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:59.255845 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.255809 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74e2a254-0bdd-452e-987b-6d2f6502e82a-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:30:59.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.977579 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" Aug 05 16:30:59.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.977597 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qnl65" event={"ID":"74e2a254-0bdd-452e-987b-6d2f6502e82a","Type":"ContainerDied","Data":"d618e5762b812b24affba20e66e6dc051cecc7966dd2175a7048557973d016e2"} Aug 05 16:30:59.977662 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.977641 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d618e5762b812b24affba20e66e6dc051cecc7966dd2175a7048557973d016e2" Aug 05 16:30:59.979807 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.979781 2567 generic.go:356] "Generic (PLEG): container finished" podID="29587b65-1f3d-4e28-8545-1bb73a761555" containerID="f3db84dca83c563c30900635eed4f99d1ab1d13340acd8e7dcf92c28d9d1e59b" exitCode=0 Aug 05 16:30:59.979916 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.979857 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" event={"ID":"29587b65-1f3d-4e28-8545-1bb73a761555","Type":"ContainerDied","Data":"f3db84dca83c563c30900635eed4f99d1ab1d13340acd8e7dcf92c28d9d1e59b"} Aug 05 16:30:59.981724 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.981707 2567 generic.go:356] "Generic (PLEG): container finished" podID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerID="5adb8fe5eabce1d376b65b0a66e09c518ac66fa43036fa5f4825e7bb6005062a" exitCode=0 Aug 05 16:30:59.981803 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:30:59.981784 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" event={"ID":"c43db9f0-9339-4f82-9a82-d98da49f5d48","Type":"ContainerDied","Data":"5adb8fe5eabce1d376b65b0a66e09c518ac66fa43036fa5f4825e7bb6005062a"} Aug 05 16:31:00.110782 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.110756 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:31:00.163925 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.163888 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-util\") pod \"47e8d448-9e64-4c06-b945-b16289a844ae\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " Aug 05 16:31:00.164145 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.164001 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-bundle\") pod \"47e8d448-9e64-4c06-b945-b16289a844ae\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " Aug 05 16:31:00.164145 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.164024 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vxms\" (UniqueName: \"kubernetes.io/projected/47e8d448-9e64-4c06-b945-b16289a844ae-kube-api-access-6vxms\") pod \"47e8d448-9e64-4c06-b945-b16289a844ae\" (UID: \"47e8d448-9e64-4c06-b945-b16289a844ae\") " Aug 05 16:31:00.164622 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.164587 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-bundle" (OuterVolumeSpecName: "bundle") pod "47e8d448-9e64-4c06-b945-b16289a844ae" (UID: "47e8d448-9e64-4c06-b945-b16289a844ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:31:00.166209 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.166182 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e8d448-9e64-4c06-b945-b16289a844ae-kube-api-access-6vxms" (OuterVolumeSpecName: "kube-api-access-6vxms") pod "47e8d448-9e64-4c06-b945-b16289a844ae" (UID: "47e8d448-9e64-4c06-b945-b16289a844ae"). InnerVolumeSpecName "kube-api-access-6vxms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:00.169525 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.169500 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-util" (OuterVolumeSpecName: "util") pod "47e8d448-9e64-4c06-b945-b16289a844ae" (UID: "47e8d448-9e64-4c06-b945-b16289a844ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:31:00.264590 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.264507 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:00.264590 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.264540 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47e8d448-9e64-4c06-b945-b16289a844ae-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:00.264590 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.264550 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxms\" (UniqueName: \"kubernetes.io/projected/47e8d448-9e64-4c06-b945-b16289a844ae-kube-api-access-6vxms\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:00.988504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.988469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" event={"ID":"47e8d448-9e64-4c06-b945-b16289a844ae","Type":"ContainerDied","Data":"24cf923977694c9600f0fc8b8e18366db1232c3342bac2b7323dabbb79361ec8"} Aug 05 16:31:00.988504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.988505 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crqsrg" Aug 05 16:31:00.989021 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:00.988525 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24cf923977694c9600f0fc8b8e18366db1232c3342bac2b7323dabbb79361ec8" Aug 05 16:31:01.114714 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.114688 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:31:01.149241 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.149216 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:31:01.170819 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.170789 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-util\") pod \"29587b65-1f3d-4e28-8545-1bb73a761555\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " Aug 05 16:31:01.171002 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.170843 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lv4m\" (UniqueName: \"kubernetes.io/projected/29587b65-1f3d-4e28-8545-1bb73a761555-kube-api-access-5lv4m\") pod \"29587b65-1f3d-4e28-8545-1bb73a761555\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " Aug 05 16:31:01.171002 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.170884 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-bundle\") pod \"29587b65-1f3d-4e28-8545-1bb73a761555\" (UID: \"29587b65-1f3d-4e28-8545-1bb73a761555\") " Aug 05 16:31:01.171812 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.171776 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-bundle" (OuterVolumeSpecName: "bundle") pod "29587b65-1f3d-4e28-8545-1bb73a761555" (UID: "29587b65-1f3d-4e28-8545-1bb73a761555"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:31:01.172994 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.172961 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29587b65-1f3d-4e28-8545-1bb73a761555-kube-api-access-5lv4m" (OuterVolumeSpecName: "kube-api-access-5lv4m") pod "29587b65-1f3d-4e28-8545-1bb73a761555" (UID: "29587b65-1f3d-4e28-8545-1bb73a761555"). InnerVolumeSpecName "kube-api-access-5lv4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:01.179277 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.179226 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-util" (OuterVolumeSpecName: "util") pod "29587b65-1f3d-4e28-8545-1bb73a761555" (UID: "29587b65-1f3d-4e28-8545-1bb73a761555"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:31:01.271764 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.271674 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-bundle\") pod \"c43db9f0-9339-4f82-9a82-d98da49f5d48\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " Aug 05 16:31:01.271764 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.271720 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpwdb\" (UniqueName: \"kubernetes.io/projected/c43db9f0-9339-4f82-9a82-d98da49f5d48-kube-api-access-gpwdb\") pod \"c43db9f0-9339-4f82-9a82-d98da49f5d48\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " Aug 05 16:31:01.271988 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.271802 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-util\") pod \"c43db9f0-9339-4f82-9a82-d98da49f5d48\" (UID: \"c43db9f0-9339-4f82-9a82-d98da49f5d48\") " Aug 05 16:31:01.271988 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.271922 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:01.271988 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.271933 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lv4m\" (UniqueName: \"kubernetes.io/projected/29587b65-1f3d-4e28-8545-1bb73a761555-kube-api-access-5lv4m\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:01.271988 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.271947 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29587b65-1f3d-4e28-8545-1bb73a761555-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:01.272241 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.272216 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-bundle" (OuterVolumeSpecName: "bundle") pod "c43db9f0-9339-4f82-9a82-d98da49f5d48" (UID: "c43db9f0-9339-4f82-9a82-d98da49f5d48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:31:01.273840 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.273817 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43db9f0-9339-4f82-9a82-d98da49f5d48-kube-api-access-gpwdb" (OuterVolumeSpecName: "kube-api-access-gpwdb") pod "c43db9f0-9339-4f82-9a82-d98da49f5d48" (UID: "c43db9f0-9339-4f82-9a82-d98da49f5d48"). InnerVolumeSpecName "kube-api-access-gpwdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:01.277416 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.277393 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-util" (OuterVolumeSpecName: "util") pod "c43db9f0-9339-4f82-9a82-d98da49f5d48" (UID: "c43db9f0-9339-4f82-9a82-d98da49f5d48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:31:01.373125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.373067 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-util\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:01.373125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.373119 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c43db9f0-9339-4f82-9a82-d98da49f5d48-bundle\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:01.373125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.373128 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwdb\" (UniqueName: \"kubernetes.io/projected/c43db9f0-9339-4f82-9a82-d98da49f5d48-kube-api-access-gpwdb\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:31:01.994447 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.994409 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" event={"ID":"29587b65-1f3d-4e28-8545-1bb73a761555","Type":"ContainerDied","Data":"923e29b3c9c105d218134ccaa946c3411ef4ddbb86300d003526f6690510d6af"} Aug 05 16:31:01.994916 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.994463 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="923e29b3c9c105d218134ccaa946c3411ef4ddbb86300d003526f6690510d6af" Aug 05 16:31:01.994916 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.994417 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kw6gb" Aug 05 16:31:01.996280 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.996256 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" event={"ID":"c43db9f0-9339-4f82-9a82-d98da49f5d48","Type":"ContainerDied","Data":"b1e5b5dc31746afafc9e6106e66230a680b25833bc176a287abce389e493ad90"} Aug 05 16:31:01.996399 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.996291 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb46xzl" Aug 05 16:31:01.996399 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:31:01.996294 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e5b5dc31746afafc9e6106e66230a680b25833bc176a287abce389e493ad90" Aug 05 16:32:48.915672 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:32:48.915650 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:32:48.916653 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:32:48.916629 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:34:21.035565 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035525 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s"] Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035902 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035920 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035939 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035947 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035957 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035966 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035979 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.035987 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036003 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036011 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036021 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036029 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036040 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036049 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036072 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036080 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="pull" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036106 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036115 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036130 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036138 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036147 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036156 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="extract" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036165 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="util" Aug 05 16:34:21.036178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036172 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="util" Aug 05 16:34:21.037234 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036245 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43db9f0-9339-4f82-9a82-d98da49f5d48" containerName="extract" Aug 05 16:34:21.037234 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036255 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="29587b65-1f3d-4e28-8545-1bb73a761555" containerName="extract" Aug 05 16:34:21.037234 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036268 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e2a254-0bdd-452e-987b-6d2f6502e82a" containerName="extract" Aug 05 16:34:21.037234 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.036277 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e8d448-9e64-4c06-b945-b16289a844ae" containerName="extract" Aug 05 16:34:21.039357 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.039335 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.041455 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.041432 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 05 16:34:21.041830 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.041817 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 05 16:34:21.041883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.041819 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g97pd\"" Aug 05 16:34:21.041883 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.041852 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 05 16:34:21.047577 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.047554 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s"] Aug 05 16:34:21.113910 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.113873 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.114117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.113944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.114117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.113976 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.114117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.113995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxxd\" (UniqueName: \"kubernetes.io/projected/2cf1912d-e93d-479e-a0a9-c08af78d005e-kube-api-access-wjxxd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.114117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.114026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.114117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.114054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf1912d-e93d-479e-a0a9-c08af78d005e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215285 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215545 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxxd\" (UniqueName: \"kubernetes.io/projected/2cf1912d-e93d-479e-a0a9-c08af78d005e-kube-api-access-wjxxd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215545 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215545 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf1912d-e93d-479e-a0a9-c08af78d005e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215705 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215547 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215782 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215842 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215827 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.215973 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.215949 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.217548 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.217525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf1912d-e93d-479e-a0a9-c08af78d005e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.217912 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.217891 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf1912d-e93d-479e-a0a9-c08af78d005e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.223199 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.223168 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxxd\" (UniqueName: \"kubernetes.io/projected/2cf1912d-e93d-479e-a0a9-c08af78d005e-kube-api-access-wjxxd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s\" (UID: \"2cf1912d-e93d-479e-a0a9-c08af78d005e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.349826 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.349793 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:21.478237 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.478212 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s"] Aug 05 16:34:21.479706 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:34:21.479679 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf1912d_e93d_479e_a0a9_c08af78d005e.slice/crio-7755d9064895e427e80ed1c6e48b1440da235b904553adbdced104d788d5512f WatchSource:0}: Error finding container 7755d9064895e427e80ed1c6e48b1440da235b904553adbdced104d788d5512f: Status 404 returned error can't find the container with id 7755d9064895e427e80ed1c6e48b1440da235b904553adbdced104d788d5512f Aug 05 16:34:21.480958 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.480939 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:34:21.893169 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:21.893129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" event={"ID":"2cf1912d-e93d-479e-a0a9-c08af78d005e","Type":"ContainerStarted","Data":"7755d9064895e427e80ed1c6e48b1440da235b904553adbdced104d788d5512f"} Aug 05 16:34:22.899447 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:22.899410 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" event={"ID":"2cf1912d-e93d-479e-a0a9-c08af78d005e","Type":"ContainerStarted","Data":"5fb8a47e42f9c920865fcfdd3db74031a55e50ed6ff37937d7246d93f8b7a905"} Aug 05 16:34:22.916414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:22.916346 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" podStartSLOduration=0.896958819 podStartE2EDuration="1.91632542s" totalImagesPullingTime="1.019366601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:34:22.915773071 +0000 UTC m=+694.375575354" watchObservedRunningTime="2026-08-05 16:34:22.91632542 +0000 UTC m=+694.376127700" Aug 05 16:34:31.350056 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:31.350025 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:31.350441 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:31.350296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:31.361959 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:31.361940 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:31.951924 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:31.951896 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s" Aug 05 16:34:43.944027 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:43.943942 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j"] Aug 05 16:34:43.946991 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:43.946971 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:43.949753 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:43.949735 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 05 16:34:43.960381 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:43.960360 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j"] Aug 05 16:34:44.107732 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.107696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.107901 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.107805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbdj\" (UniqueName: \"kubernetes.io/projected/16f37893-5aa0-4980-9da5-907fa48c2480-kube-api-access-trbdj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.107901 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.107834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.107901 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.107861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.107901 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.107888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.108053 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.107918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f37893-5aa0-4980-9da5-907fa48c2480-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.208793 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.208708 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.208793 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.208778 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbdj\" (UniqueName: \"kubernetes.io/projected/16f37893-5aa0-4980-9da5-907fa48c2480-kube-api-access-trbdj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.208999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.208803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.208999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.208829 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.208999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.208855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.208999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.208884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f37893-5aa0-4980-9da5-907fa48c2480-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.209245 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.209217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.209245 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.209227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.209333 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.209290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.211061 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.211040 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16f37893-5aa0-4980-9da5-907fa48c2480-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.211345 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.211329 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16f37893-5aa0-4980-9da5-907fa48c2480-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.215983 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.215962 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbdj\" (UniqueName: \"kubernetes.io/projected/16f37893-5aa0-4980-9da5-907fa48c2480-kube-api-access-trbdj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ck92j\" (UID: \"16f37893-5aa0-4980-9da5-907fa48c2480\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.256603 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.256581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:44.380959 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.380936 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j"] Aug 05 16:34:44.383416 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:34:44.383389 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f37893_5aa0_4980_9da5_907fa48c2480.slice/crio-9f5aa9dfdcd141145bf0215a065832715f6b8df1e04adf6f38b9523a3880d862 WatchSource:0}: Error finding container 9f5aa9dfdcd141145bf0215a065832715f6b8df1e04adf6f38b9523a3880d862: Status 404 returned error can't find the container with id 9f5aa9dfdcd141145bf0215a065832715f6b8df1e04adf6f38b9523a3880d862 Aug 05 16:34:44.997804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.997708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" event={"ID":"16f37893-5aa0-4980-9da5-907fa48c2480","Type":"ContainerStarted","Data":"5d237145f2e01fb32549dc9e12af808b76314b0db588f611de3aea8b70116037"} Aug 05 16:34:44.997804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:44.997769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" event={"ID":"16f37893-5aa0-4980-9da5-907fa48c2480","Type":"ContainerStarted","Data":"9f5aa9dfdcd141145bf0215a065832715f6b8df1e04adf6f38b9523a3880d862"} Aug 05 16:34:45.013792 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:45.013748 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" podStartSLOduration=1.677938972 podStartE2EDuration="2.013731064s" totalImagesPullingTime="335.792092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:34:45.012827804 +0000 UTC m=+716.472630096" watchObservedRunningTime="2026-08-05 16:34:45.013731064 +0000 UTC m=+716.473533342" Aug 05 16:34:54.257055 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:54.257018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:54.257600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:54.257111 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:54.269549 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:54.269525 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:34:55.051033 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:34:55.051006 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ck92j" Aug 05 16:35:19.548031 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.547995 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj"] Aug 05 16:35:19.551742 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.551720 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.553541 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.553516 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 05 16:35:19.560859 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.560837 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj"] Aug 05 16:35:19.568983 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.568960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.569117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.569010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936a1fd1-b818-429e-8551-9f3bd3afaf47-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.569117 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.569079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94g9\" (UniqueName: \"kubernetes.io/projected/936a1fd1-b818-429e-8551-9f3bd3afaf47-kube-api-access-f94g9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.569242 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.569148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.569242 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.569177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.569242 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.569193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670040 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936a1fd1-b818-429e-8551-9f3bd3afaf47-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670265 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94g9\" (UniqueName: \"kubernetes.io/projected/936a1fd1-b818-429e-8551-9f3bd3afaf47-kube-api-access-f94g9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670265 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670265 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670265 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670181 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670265 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670581 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670543 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670647 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.670749 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.670726 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.672805 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.672777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936a1fd1-b818-429e-8551-9f3bd3afaf47-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.672961 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.672927 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936a1fd1-b818-429e-8551-9f3bd3afaf47-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.678582 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.678549 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94g9\" (UniqueName: \"kubernetes.io/projected/936a1fd1-b818-429e-8551-9f3bd3afaf47-kube-api-access-f94g9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-skwzj\" (UID: \"936a1fd1-b818-429e-8551-9f3bd3afaf47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.862142 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.862080 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:19.990946 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:19.990790 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj"] Aug 05 16:35:19.993244 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:35:19.993218 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936a1fd1_b818_429e_8551_9f3bd3afaf47.slice/crio-5e490abbf9a7a36975292d702dd13e870888a5a1f53a633ffc0f3d370945f3ee WatchSource:0}: Error finding container 5e490abbf9a7a36975292d702dd13e870888a5a1f53a633ffc0f3d370945f3ee: Status 404 returned error can't find the container with id 5e490abbf9a7a36975292d702dd13e870888a5a1f53a633ffc0f3d370945f3ee Aug 05 16:35:20.154099 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:20.154000 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" event={"ID":"936a1fd1-b818-429e-8551-9f3bd3afaf47","Type":"ContainerStarted","Data":"5e490abbf9a7a36975292d702dd13e870888a5a1f53a633ffc0f3d370945f3ee"} Aug 05 16:35:21.159602 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:21.159562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" event={"ID":"936a1fd1-b818-429e-8551-9f3bd3afaf47","Type":"ContainerStarted","Data":"2cbeef07c06413a61aa0462d638e18931c920e09956665285404ade4b7d0c641"} Aug 05 16:35:21.175667 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:21.175614 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" podStartSLOduration=1.951080887 podStartE2EDuration="2.17559653s" totalImagesPullingTime="224.515643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:35:21.174031061 +0000 UTC m=+752.633833354" watchObservedRunningTime="2026-08-05 16:35:21.17559653 +0000 UTC m=+752.635398810" Aug 05 16:35:29.863064 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:29.863027 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:29.863570 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:29.863141 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:29.875476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:29.875452 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:35:30.215727 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:35:30.215647 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-skwzj" Aug 05 16:37:48.944190 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:37:48.944116 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:37:48.946482 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:37:48.946460 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:42:48.972943 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:42:48.972912 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:42:48.975656 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:42:48.975632 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:47:49.007842 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:47:49.007812 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:47:49.018631 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:47:49.018607 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:52:49.043870 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:52:49.043841 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:52:49.049641 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:52:49.049623 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:54:30.671544 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.671467 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4"] Aug 05 16:54:30.674872 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.674852 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.677241 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.677215 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 16:54:30.677375 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.677333 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:54:30.677614 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.677595 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b701e0ce-serving-cert\"" Aug 05 16:54:30.677862 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.677846 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 16:54:30.695150 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.689882 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4"] Aug 05 16:54:30.859771 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.859726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-maas-api-tls\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.859973 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.859799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-openshift-service-ca\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.859973 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.859852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcn57\" (UniqueName: \"kubernetes.io/projected/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-kube-api-access-qcn57\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.960695 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.960592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-openshift-service-ca\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.960695 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.960661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcn57\" (UniqueName: \"kubernetes.io/projected/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-kube-api-access-qcn57\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.960934 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.960706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-maas-api-tls\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.961341 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.961319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-openshift-service-ca\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.963188 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.963167 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-maas-api-tls\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.969666 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.969640 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcn57\" (UniqueName: \"kubernetes.io/projected/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-kube-api-access-qcn57\") pod \"maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:30.985505 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:30.985477 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:31.124314 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:31.124286 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4"] Aug 05 16:54:31.126502 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:54:31.126465 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebac2f94_cd95_4cf6_9191_7aa7e6cc471c.slice/crio-f99b8d46e13d52785c54542671086e99dab6d2a92469029eed5f99a563c94c92 WatchSource:0}: Error finding container f99b8d46e13d52785c54542671086e99dab6d2a92469029eed5f99a563c94c92: Status 404 returned error can't find the container with id f99b8d46e13d52785c54542671086e99dab6d2a92469029eed5f99a563c94c92 Aug 05 16:54:31.127894 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:31.127872 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:54:31.543162 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:31.543124 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" event={"ID":"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c","Type":"ContainerStarted","Data":"f99b8d46e13d52785c54542671086e99dab6d2a92469029eed5f99a563c94c92"} Aug 05 16:54:34.560722 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:34.560684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" event={"ID":"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c","Type":"ContainerStarted","Data":"c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031"} Aug 05 16:54:34.561128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:34.560746 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:34.581548 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:34.581497 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" podStartSLOduration=2.179048614 podStartE2EDuration="4.581482313s" totalImagesPullingTime="2.402433699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:54:34.579814979 +0000 UTC m=+1906.039617259" watchObservedRunningTime="2026-08-05 16:54:34.581482313 +0000 UTC m=+1906.041284593" Aug 05 16:54:40.571582 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:40.571548 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:49.819301 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:49.819267 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4"] Aug 05 16:54:49.819759 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:49.819481 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" podUID="ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" containerName="maas-api" containerID="cri-o://c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031" gracePeriod=30 Aug 05 16:54:50.064205 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.064181 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:50.114119 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.114066 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcn57\" (UniqueName: \"kubernetes.io/projected/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-kube-api-access-qcn57\") pod \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " Aug 05 16:54:50.114306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.114182 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-maas-api-tls\") pod \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " Aug 05 16:54:50.114306 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.114216 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-openshift-service-ca\") pod \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\" (UID: \"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c\") " Aug 05 16:54:50.114681 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.114650 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" (UID: "ebac2f94-cd95-4cf6-9191-7aa7e6cc471c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:54:50.116322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.116294 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" (UID: "ebac2f94-cd95-4cf6-9191-7aa7e6cc471c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:54:50.116410 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.116348 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-kube-api-access-qcn57" (OuterVolumeSpecName: "kube-api-access-qcn57") pod "ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" (UID: "ebac2f94-cd95-4cf6-9191-7aa7e6cc471c"). InnerVolumeSpecName "kube-api-access-qcn57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:54:50.215395 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.215355 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:54:50.215395 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.215392 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:54:50.215633 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.215406 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcn57\" (UniqueName: \"kubernetes.io/projected/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c-kube-api-access-qcn57\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:54:50.642411 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.642377 2567 generic.go:356] "Generic (PLEG): container finished" podID="ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" containerID="c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031" exitCode=0 Aug 05 16:54:50.642411 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.642413 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" Aug 05 16:54:50.642624 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.642456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" event={"ID":"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c","Type":"ContainerDied","Data":"c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031"} Aug 05 16:54:50.642624 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.642508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4" event={"ID":"ebac2f94-cd95-4cf6-9191-7aa7e6cc471c","Type":"ContainerDied","Data":"f99b8d46e13d52785c54542671086e99dab6d2a92469029eed5f99a563c94c92"} Aug 05 16:54:50.642624 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.642530 2567 scope.go:117] "RemoveContainer" containerID="c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031" Aug 05 16:54:50.651840 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.651826 2567 scope.go:117] "RemoveContainer" containerID="c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031" Aug 05 16:54:50.652069 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:54:50.652051 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031\": container with ID starting with c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031 not found: ID does not exist" containerID="c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031" Aug 05 16:54:50.652156 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.652078 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031"} err="failed to get container status \"c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031\": rpc error: code = NotFound desc = could not find container \"c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031\": container with ID starting with c96c7fcc0633611c3723c97fa2dcfed5df78a4f1ba5f1a43f18c8cb18c7e9031 not found: ID does not exist" Aug 05 16:54:50.670815 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.670788 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4"] Aug 05 16:54:50.675894 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:50.675872 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b701e0ce-844d7d4f7c-ffbg4"] Aug 05 16:54:51.010974 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:51.010891 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" path="/var/lib/kubelet/pods/ebac2f94-cd95-4cf6-9191-7aa7e6cc471c/volumes" Aug 05 16:54:54.571949 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.571908 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc"] Aug 05 16:54:54.572419 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.572402 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" containerName="maas-api" Aug 05 16:54:54.572468 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.572422 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" containerName="maas-api" Aug 05 16:54:54.572504 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.572494 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebac2f94-cd95-4cf6-9191-7aa7e6cc471c" containerName="maas-api" Aug 05 16:54:54.575462 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.575438 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.578759 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.578738 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dfc96bfb-serving-cert\"" Aug 05 16:54:54.578984 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.578970 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 16:54:54.579097 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.579065 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 16:54:54.579310 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.579295 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:54:54.594144 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.594115 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc"] Aug 05 16:54:54.653214 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.653177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505f752c-b2dc-4a88-8f66-c2591db32bf7-openshift-service-ca\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.653214 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.653216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.653430 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.653245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts584\" (UniqueName: \"kubernetes.io/projected/505f752c-b2dc-4a88-8f66-c2591db32bf7-kube-api-access-ts584\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.754511 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.754475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505f752c-b2dc-4a88-8f66-c2591db32bf7-openshift-service-ca\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.754694 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.754520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.754694 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.754540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts584\" (UniqueName: \"kubernetes.io/projected/505f752c-b2dc-4a88-8f66-c2591db32bf7-kube-api-access-ts584\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.754694 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:54:54.754640 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-serving-cert: secret "maas-api-e2e-migrate-dfc96bfb-serving-cert" not found Aug 05 16:54:54.754805 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:54:54.754721 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls podName:505f752c-b2dc-4a88-8f66-c2591db32bf7 nodeName:}" failed. No retries permitted until 2026-08-05 16:54:55.254699087 +0000 UTC m=+1926.714501346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls") pod "maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" (UID: "505f752c-b2dc-4a88-8f66-c2591db32bf7") : secret "maas-api-e2e-migrate-dfc96bfb-serving-cert" not found Aug 05 16:54:54.755175 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.755157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505f752c-b2dc-4a88-8f66-c2591db32bf7-openshift-service-ca\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:54.763561 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:54.763532 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts584\" (UniqueName: \"kubernetes.io/projected/505f752c-b2dc-4a88-8f66-c2591db32bf7-kube-api-access-ts584\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:55.260192 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.260153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:55.262713 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.262688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls\") pod \"maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:55.352937 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.352895 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9"] Aug 05 16:54:55.356055 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.356033 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.358794 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.358761 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-dfc96bfb\"" Aug 05 16:54:55.358915 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.358807 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-dfc96bfb-dockercfg-g4d72\"" Aug 05 16:54:55.374732 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.374705 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9"] Aug 05 16:54:55.461145 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.461068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5dc\" (UniqueName: \"kubernetes.io/projected/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-kube-api-access-2k5dc\") pod \"payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.461354 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.461256 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.487633 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.487594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:55.561776 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.561679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.561776 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.561745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5dc\" (UniqueName: \"kubernetes.io/projected/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-kube-api-access-2k5dc\") pod \"payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.562411 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.562391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.572722 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.572696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5dc\" (UniqueName: \"kubernetes.io/projected/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-kube-api-access-2k5dc\") pod \"payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.627358 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.627327 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc"] Aug 05 16:54:55.628837 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:54:55.628812 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505f752c_b2dc_4a88_8f66_c2591db32bf7.slice/crio-a6b99c8305ef5b970dda2bf6b840edf8d7dd22ce3f2e7a63151b539bac8c84cc WatchSource:0}: Error finding container a6b99c8305ef5b970dda2bf6b840edf8d7dd22ce3f2e7a63151b539bac8c84cc: Status 404 returned error can't find the container with id a6b99c8305ef5b970dda2bf6b840edf8d7dd22ce3f2e7a63151b539bac8c84cc Aug 05 16:54:55.666173 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.666148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:55.674138 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.674112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" event={"ID":"505f752c-b2dc-4a88-8f66-c2591db32bf7","Type":"ContainerStarted","Data":"a6b99c8305ef5b970dda2bf6b840edf8d7dd22ce3f2e7a63151b539bac8c84cc"} Aug 05 16:54:55.806423 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:55.806389 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9"] Aug 05 16:54:55.808231 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:54:55.808202 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f110e9_6e7e_4cc8_9177_c1033b39dab8.slice/crio-7bc10127aa6ef5c9d305b0d37eaf236a6573bb6204d7279061f2a4ce85be1c39 WatchSource:0}: Error finding container 7bc10127aa6ef5c9d305b0d37eaf236a6573bb6204d7279061f2a4ce85be1c39: Status 404 returned error can't find the container with id 7bc10127aa6ef5c9d305b0d37eaf236a6573bb6204d7279061f2a4ce85be1c39 Aug 05 16:54:56.682130 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:56.682067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" event={"ID":"505f752c-b2dc-4a88-8f66-c2591db32bf7","Type":"ContainerStarted","Data":"5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6"} Aug 05 16:54:56.682627 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:56.682298 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:54:56.683559 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:56.683533 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" event={"ID":"e7f110e9-6e7e-4cc8-9177-c1033b39dab8","Type":"ContainerStarted","Data":"7bc10127aa6ef5c9d305b0d37eaf236a6573bb6204d7279061f2a4ce85be1c39"} Aug 05 16:54:56.702823 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:56.702751 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" podStartSLOduration=2.37552061 podStartE2EDuration="2.702734412s" totalImagesPullingTime="327.213802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:54:56.700875962 +0000 UTC m=+1928.160678242" watchObservedRunningTime="2026-08-05 16:54:56.702734412 +0000 UTC m=+1928.162536694" Aug 05 16:54:57.689480 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:57.689445 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" event={"ID":"e7f110e9-6e7e-4cc8-9177-c1033b39dab8","Type":"ContainerStarted","Data":"83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa"} Aug 05 16:54:57.689892 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:57.689587 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:54:57.715636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:54:57.715583 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" podStartSLOduration=0.916850555 podStartE2EDuration="2.715566424s" totalImagesPullingTime="1.798715869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:54:57.713648979 +0000 UTC m=+1929.173451260" watchObservedRunningTime="2026-08-05 16:54:57.715566424 +0000 UTC m=+1929.175368774" Aug 05 16:55:02.695458 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:02.695430 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:55:08.696670 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:08.696634 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:55:25.401935 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.401893 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc"] Aug 05 16:55:25.402813 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.402768 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" podUID="505f752c-b2dc-4a88-8f66-c2591db32bf7" containerName="maas-api" containerID="cri-o://5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6" gracePeriod=30 Aug 05 16:55:25.485061 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.484813 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9"] Aug 05 16:55:25.485292 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.485055 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" podUID="e7f110e9-6e7e-4cc8-9177-c1033b39dab8" containerName="payload-processing" containerID="cri-o://83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa" gracePeriod=30 Aug 05 16:55:25.729806 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.729778 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:55:25.737230 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.737204 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505f752c-b2dc-4a88-8f66-c2591db32bf7-openshift-service-ca\") pod \"505f752c-b2dc-4a88-8f66-c2591db32bf7\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " Aug 05 16:55:25.737367 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.737249 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls\") pod \"505f752c-b2dc-4a88-8f66-c2591db32bf7\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " Aug 05 16:55:25.737367 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.737272 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts584\" (UniqueName: \"kubernetes.io/projected/505f752c-b2dc-4a88-8f66-c2591db32bf7-kube-api-access-ts584\") pod \"505f752c-b2dc-4a88-8f66-c2591db32bf7\" (UID: \"505f752c-b2dc-4a88-8f66-c2591db32bf7\") " Aug 05 16:55:25.737581 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.737554 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505f752c-b2dc-4a88-8f66-c2591db32bf7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "505f752c-b2dc-4a88-8f66-c2591db32bf7" (UID: "505f752c-b2dc-4a88-8f66-c2591db32bf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:55:25.739373 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.739348 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "505f752c-b2dc-4a88-8f66-c2591db32bf7" (UID: "505f752c-b2dc-4a88-8f66-c2591db32bf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:55:25.739485 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.739379 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505f752c-b2dc-4a88-8f66-c2591db32bf7-kube-api-access-ts584" (OuterVolumeSpecName: "kube-api-access-ts584") pod "505f752c-b2dc-4a88-8f66-c2591db32bf7" (UID: "505f752c-b2dc-4a88-8f66-c2591db32bf7"). InnerVolumeSpecName "kube-api-access-ts584". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:55:25.754718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.754695 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:55:25.827890 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.827856 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7f110e9-6e7e-4cc8-9177-c1033b39dab8" containerID="83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa" exitCode=0 Aug 05 16:55:25.828076 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.827900 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" Aug 05 16:55:25.828076 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.827951 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" event={"ID":"e7f110e9-6e7e-4cc8-9177-c1033b39dab8","Type":"ContainerDied","Data":"83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa"} Aug 05 16:55:25.828076 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.828013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9" event={"ID":"e7f110e9-6e7e-4cc8-9177-c1033b39dab8","Type":"ContainerDied","Data":"7bc10127aa6ef5c9d305b0d37eaf236a6573bb6204d7279061f2a4ce85be1c39"} Aug 05 16:55:25.828076 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.828046 2567 scope.go:117] "RemoveContainer" containerID="83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa" Aug 05 16:55:25.829328 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.829310 2567 generic.go:356] "Generic (PLEG): container finished" podID="505f752c-b2dc-4a88-8f66-c2591db32bf7" containerID="5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6" exitCode=0 Aug 05 16:55:25.829404 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.829343 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" event={"ID":"505f752c-b2dc-4a88-8f66-c2591db32bf7","Type":"ContainerDied","Data":"5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6"} Aug 05 16:55:25.829404 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.829365 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" Aug 05 16:55:25.829404 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.829383 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc" event={"ID":"505f752c-b2dc-4a88-8f66-c2591db32bf7","Type":"ContainerDied","Data":"a6b99c8305ef5b970dda2bf6b840edf8d7dd22ce3f2e7a63151b539bac8c84cc"} Aug 05 16:55:25.837718 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.837693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-plugins-config-volume\") pod \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " Aug 05 16:55:25.837864 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.837731 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k5dc\" (UniqueName: \"kubernetes.io/projected/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-kube-api-access-2k5dc\") pod \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\" (UID: \"e7f110e9-6e7e-4cc8-9177-c1033b39dab8\") " Aug 05 16:55:25.838099 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.838067 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505f752c-b2dc-4a88-8f66-c2591db32bf7-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:55:25.838186 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.838113 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505f752c-b2dc-4a88-8f66-c2591db32bf7-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:55:25.838186 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.838128 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts584\" (UniqueName: \"kubernetes.io/projected/505f752c-b2dc-4a88-8f66-c2591db32bf7-kube-api-access-ts584\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:55:25.838186 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.838131 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7f110e9-6e7e-4cc8-9177-c1033b39dab8" (UID: "e7f110e9-6e7e-4cc8-9177-c1033b39dab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:55:25.839841 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.839760 2567 scope.go:117] "RemoveContainer" containerID="83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa" Aug 05 16:55:25.840325 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:55:25.840081 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa\": container with ID starting with 83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa not found: ID does not exist" containerID="83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa" Aug 05 16:55:25.840325 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.840166 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa"} err="failed to get container status \"83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa\": rpc error: code = NotFound desc = could not find container \"83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa\": container with ID starting with 83e303cf83d3f1af02be8dbc4f2b8797f63eeb8240e34f46d6de85598e9b9daa not found: ID does not exist" Aug 05 16:55:25.840325 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.840208 2567 scope.go:117] "RemoveContainer" containerID="5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6" Aug 05 16:55:25.842628 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.842599 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-kube-api-access-2k5dc" (OuterVolumeSpecName: "kube-api-access-2k5dc") pod "e7f110e9-6e7e-4cc8-9177-c1033b39dab8" (UID: "e7f110e9-6e7e-4cc8-9177-c1033b39dab8"). InnerVolumeSpecName "kube-api-access-2k5dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:55:25.849390 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.849367 2567 scope.go:117] "RemoveContainer" containerID="5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6" Aug 05 16:55:25.849698 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:55:25.849677 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6\": container with ID starting with 5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6 not found: ID does not exist" containerID="5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6" Aug 05 16:55:25.849767 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.849705 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6"} err="failed to get container status \"5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6\": rpc error: code = NotFound desc = could not find container \"5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6\": container with ID starting with 5a3a392cab2401f7de75de83750c134063ebe60f25b946ca9b4980a4340162f6 not found: ID does not exist" Aug 05 16:55:25.868753 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.868721 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc"] Aug 05 16:55:25.872934 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.872905 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dfc96bfb-757c589cdf-thlnc"] Aug 05 16:55:25.938947 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.938898 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:55:25.938947 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:25.938941 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5dc\" (UniqueName: \"kubernetes.io/projected/e7f110e9-6e7e-4cc8-9177-c1033b39dab8-kube-api-access-2k5dc\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:55:26.153137 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:26.153103 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9"] Aug 05 16:55:26.157693 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:26.157663 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dfc96bfb-576bdc7769-n85c9"] Aug 05 16:55:27.011024 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:27.010990 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505f752c-b2dc-4a88-8f66-c2591db32bf7" path="/var/lib/kubelet/pods/505f752c-b2dc-4a88-8f66-c2591db32bf7/volumes" Aug 05 16:55:27.011433 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:27.011387 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f110e9-6e7e-4cc8-9177-c1033b39dab8" path="/var/lib/kubelet/pods/e7f110e9-6e7e-4cc8-9177-c1033b39dab8/volumes" Aug 05 16:55:30.600305 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600266 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44"] Aug 05 16:55:30.600789 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600657 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505f752c-b2dc-4a88-8f66-c2591db32bf7" containerName="maas-api" Aug 05 16:55:30.600789 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600673 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="505f752c-b2dc-4a88-8f66-c2591db32bf7" containerName="maas-api" Aug 05 16:55:30.600789 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600690 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f110e9-6e7e-4cc8-9177-c1033b39dab8" containerName="payload-processing" Aug 05 16:55:30.600789 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600697 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f110e9-6e7e-4cc8-9177-c1033b39dab8" containerName="payload-processing" Aug 05 16:55:30.600789 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600759 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="505f752c-b2dc-4a88-8f66-c2591db32bf7" containerName="maas-api" Aug 05 16:55:30.600789 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.600767 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f110e9-6e7e-4cc8-9177-c1033b39dab8" containerName="payload-processing" Aug 05 16:55:30.603715 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.603697 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.605915 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.605889 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 16:55:30.606974 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.606953 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:55:30.607104 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.606953 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 16:55:30.607104 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.606957 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3a6cf6a5-serving-cert\"" Aug 05 16:55:30.618591 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.618565 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44"] Aug 05 16:55:30.672810 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.672771 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c347e886-9247-42dc-a4b7-ffb5340b6574-maas-api-tls\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.672981 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.672827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347e886-9247-42dc-a4b7-ffb5340b6574-openshift-service-ca\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.672981 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.672858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drk4c\" (UniqueName: \"kubernetes.io/projected/c347e886-9247-42dc-a4b7-ffb5340b6574-kube-api-access-drk4c\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.773361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.773325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347e886-9247-42dc-a4b7-ffb5340b6574-openshift-service-ca\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.773536 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.773370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drk4c\" (UniqueName: \"kubernetes.io/projected/c347e886-9247-42dc-a4b7-ffb5340b6574-kube-api-access-drk4c\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.773536 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.773434 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c347e886-9247-42dc-a4b7-ffb5340b6574-maas-api-tls\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.774066 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.774027 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347e886-9247-42dc-a4b7-ffb5340b6574-openshift-service-ca\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.775812 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.775782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c347e886-9247-42dc-a4b7-ffb5340b6574-maas-api-tls\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.783878 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.783858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drk4c\" (UniqueName: \"kubernetes.io/projected/c347e886-9247-42dc-a4b7-ffb5340b6574-kube-api-access-drk4c\") pod \"maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:30.914364 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:30.914273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:31.050785 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.050740 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44"] Aug 05 16:55:31.053054 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:55:31.053030 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc347e886_9247_42dc_a4b7_ffb5340b6574.slice/crio-739f9b1d4a7e158971bae1e6b57b408d893234f08b528b7e58c3ec3a00588f43 WatchSource:0}: Error finding container 739f9b1d4a7e158971bae1e6b57b408d893234f08b528b7e58c3ec3a00588f43: Status 404 returned error can't find the container with id 739f9b1d4a7e158971bae1e6b57b408d893234f08b528b7e58c3ec3a00588f43 Aug 05 16:55:31.579896 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.579860 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r"] Aug 05 16:55:31.582166 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.582149 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.584530 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.584507 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3a6cf6a5-dockercfg-9l7fn\"" Aug 05 16:55:31.584648 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.584578 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3a6cf6a5\"" Aug 05 16:55:31.666277 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.666244 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj"] Aug 05 16:55:31.668564 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.668536 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r"] Aug 05 16:55:31.668564 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.668564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.680314 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.680291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbv9\" (UniqueName: \"kubernetes.io/projected/a33b4db7-471f-422c-9b86-68c27ccec8ba-kube-api-access-kdbv9\") pod \"payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.680446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.680333 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43866555-48c4-49e6-a026-f0382fed7525-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.680446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.680395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33b4db7-471f-422c-9b86-68c27ccec8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.680589 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.680459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhqf\" (UniqueName: \"kubernetes.io/projected/43866555-48c4-49e6-a026-f0382fed7525-kube-api-access-nwhqf\") pod \"payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.690811 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.690782 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj"] Aug 05 16:55:31.781348 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.781268 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33b4db7-471f-422c-9b86-68c27ccec8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.781348 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.781330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhqf\" (UniqueName: \"kubernetes.io/projected/43866555-48c4-49e6-a026-f0382fed7525-kube-api-access-nwhqf\") pod \"payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.781604 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.781364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbv9\" (UniqueName: \"kubernetes.io/projected/a33b4db7-471f-422c-9b86-68c27ccec8ba-kube-api-access-kdbv9\") pod \"payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.781604 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.781430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43866555-48c4-49e6-a026-f0382fed7525-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.782336 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.782311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43866555-48c4-49e6-a026-f0382fed7525-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.782391 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.782335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33b4db7-471f-422c-9b86-68c27ccec8ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.790681 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.790651 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbv9\" (UniqueName: \"kubernetes.io/projected/a33b4db7-471f-422c-9b86-68c27ccec8ba-kube-api-access-kdbv9\") pod \"payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.790788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.790763 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhqf\" (UniqueName: \"kubernetes.io/projected/43866555-48c4-49e6-a026-f0382fed7525-kube-api-access-nwhqf\") pod \"payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:31.861446 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.861414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" event={"ID":"c347e886-9247-42dc-a4b7-ffb5340b6574","Type":"ContainerStarted","Data":"76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09"} Aug 05 16:55:31.861623 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.861466 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" event={"ID":"c347e886-9247-42dc-a4b7-ffb5340b6574","Type":"ContainerStarted","Data":"739f9b1d4a7e158971bae1e6b57b408d893234f08b528b7e58c3ec3a00588f43"} Aug 05 16:55:31.861623 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.861529 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:31.879925 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.879855 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" podStartSLOduration=1.485576219 podStartE2EDuration="1.879833424s" totalImagesPullingTime="394.257205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:55:31.879246204 +0000 UTC m=+1963.339048485" watchObservedRunningTime="2026-08-05 16:55:31.879833424 +0000 UTC m=+1963.339635706" Aug 05 16:55:31.891775 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.891746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:31.980455 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:31.980423 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:32.026841 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.026808 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r"] Aug 05 16:55:32.028465 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:55:32.028439 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33b4db7_471f_422c_9b86_68c27ccec8ba.slice/crio-413607b5923a77ae75c6987934b519ca70bbc31dca5925fe4852c14cff7fd2d2 WatchSource:0}: Error finding container 413607b5923a77ae75c6987934b519ca70bbc31dca5925fe4852c14cff7fd2d2: Status 404 returned error can't find the container with id 413607b5923a77ae75c6987934b519ca70bbc31dca5925fe4852c14cff7fd2d2 Aug 05 16:55:32.125773 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.125730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj"] Aug 05 16:55:32.127877 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:55:32.127847 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43866555_48c4_49e6_a026_f0382fed7525.slice/crio-10e3866d59f454122a106b3fe63f0618594f232c5b832ceb1d22f64d7ca20670 WatchSource:0}: Error finding container 10e3866d59f454122a106b3fe63f0618594f232c5b832ceb1d22f64d7ca20670: Status 404 returned error can't find the container with id 10e3866d59f454122a106b3fe63f0618594f232c5b832ceb1d22f64d7ca20670 Aug 05 16:55:32.867575 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.867529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" event={"ID":"a33b4db7-471f-422c-9b86-68c27ccec8ba","Type":"ContainerStarted","Data":"b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1"} Aug 05 16:55:32.868125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.867604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" event={"ID":"a33b4db7-471f-422c-9b86-68c27ccec8ba","Type":"ContainerStarted","Data":"413607b5923a77ae75c6987934b519ca70bbc31dca5925fe4852c14cff7fd2d2"} Aug 05 16:55:32.868125 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.867723 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:32.869209 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.869184 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" event={"ID":"43866555-48c4-49e6-a026-f0382fed7525","Type":"ContainerStarted","Data":"ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40"} Aug 05 16:55:32.869336 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.869224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" event={"ID":"43866555-48c4-49e6-a026-f0382fed7525","Type":"ContainerStarted","Data":"10e3866d59f454122a106b3fe63f0618594f232c5b832ceb1d22f64d7ca20670"} Aug 05 16:55:32.869382 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.869340 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:55:32.885881 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.885835 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" podStartSLOduration=1.88581991 podStartE2EDuration="1.88581991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:55:32.884445303 +0000 UTC m=+1964.344247584" watchObservedRunningTime="2026-08-05 16:55:32.88581991 +0000 UTC m=+1964.345622234" Aug 05 16:55:32.905033 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:32.904975 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" podStartSLOduration=1.904955505 podStartE2EDuration="1.904955505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:55:32.903493666 +0000 UTC m=+1964.363295961" watchObservedRunningTime="2026-08-05 16:55:32.904955505 +0000 UTC m=+1964.364757786" Aug 05 16:55:37.874668 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:37.874633 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:55:43.877634 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:43.877546 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:55:43.878044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:55:43.877665 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:56:01.272658 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.272625 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44"] Aug 05 16:56:01.274237 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.272834 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" podUID="c347e886-9247-42dc-a4b7-ffb5340b6574" containerName="maas-api" containerID="cri-o://76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09" gracePeriod=30 Aug 05 16:56:01.382905 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.382869 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj"] Aug 05 16:56:01.383169 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.383141 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" podUID="43866555-48c4-49e6-a026-f0382fed7525" containerName="payload-processing" containerID="cri-o://ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40" gracePeriod=30 Aug 05 16:56:01.397079 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.397048 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r"] Aug 05 16:56:01.397324 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.397282 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" podUID="a33b4db7-471f-422c-9b86-68c27ccec8ba" containerName="payload-pre-processing" containerID="cri-o://b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1" gracePeriod=30 Aug 05 16:56:01.529861 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.529789 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:56:01.643034 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.642996 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c347e886-9247-42dc-a4b7-ffb5340b6574-maas-api-tls\") pod \"c347e886-9247-42dc-a4b7-ffb5340b6574\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " Aug 05 16:56:01.643034 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.643037 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drk4c\" (UniqueName: \"kubernetes.io/projected/c347e886-9247-42dc-a4b7-ffb5340b6574-kube-api-access-drk4c\") pod \"c347e886-9247-42dc-a4b7-ffb5340b6574\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " Aug 05 16:56:01.643330 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.643136 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347e886-9247-42dc-a4b7-ffb5340b6574-openshift-service-ca\") pod \"c347e886-9247-42dc-a4b7-ffb5340b6574\" (UID: \"c347e886-9247-42dc-a4b7-ffb5340b6574\") " Aug 05 16:56:01.643591 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.643564 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347e886-9247-42dc-a4b7-ffb5340b6574-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c347e886-9247-42dc-a4b7-ffb5340b6574" (UID: "c347e886-9247-42dc-a4b7-ffb5340b6574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:01.645478 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.645448 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c347e886-9247-42dc-a4b7-ffb5340b6574-kube-api-access-drk4c" (OuterVolumeSpecName: "kube-api-access-drk4c") pod "c347e886-9247-42dc-a4b7-ffb5340b6574" (UID: "c347e886-9247-42dc-a4b7-ffb5340b6574"). InnerVolumeSpecName "kube-api-access-drk4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:01.645904 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.645885 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c347e886-9247-42dc-a4b7-ffb5340b6574-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c347e886-9247-42dc-a4b7-ffb5340b6574" (UID: "c347e886-9247-42dc-a4b7-ffb5340b6574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:56:01.653463 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.653440 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:56:01.660762 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.660743 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:56:01.744343 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744307 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwhqf\" (UniqueName: \"kubernetes.io/projected/43866555-48c4-49e6-a026-f0382fed7525-kube-api-access-nwhqf\") pod \"43866555-48c4-49e6-a026-f0382fed7525\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " Aug 05 16:56:01.744343 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744350 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33b4db7-471f-422c-9b86-68c27ccec8ba-plugins-config-volume\") pod \"a33b4db7-471f-422c-9b86-68c27ccec8ba\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " Aug 05 16:56:01.744593 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744377 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdbv9\" (UniqueName: \"kubernetes.io/projected/a33b4db7-471f-422c-9b86-68c27ccec8ba-kube-api-access-kdbv9\") pod \"a33b4db7-471f-422c-9b86-68c27ccec8ba\" (UID: \"a33b4db7-471f-422c-9b86-68c27ccec8ba\") " Aug 05 16:56:01.744593 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744458 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43866555-48c4-49e6-a026-f0382fed7525-plugins-config-volume\") pod \"43866555-48c4-49e6-a026-f0382fed7525\" (UID: \"43866555-48c4-49e6-a026-f0382fed7525\") " Aug 05 16:56:01.744708 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744638 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347e886-9247-42dc-a4b7-ffb5340b6574-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:01.744708 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744656 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c347e886-9247-42dc-a4b7-ffb5340b6574-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:01.744708 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744670 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drk4c\" (UniqueName: \"kubernetes.io/projected/c347e886-9247-42dc-a4b7-ffb5340b6574-kube-api-access-drk4c\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:01.744839 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744769 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33b4db7-471f-422c-9b86-68c27ccec8ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a33b4db7-471f-422c-9b86-68c27ccec8ba" (UID: "a33b4db7-471f-422c-9b86-68c27ccec8ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:01.744839 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.744794 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43866555-48c4-49e6-a026-f0382fed7525-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43866555-48c4-49e6-a026-f0382fed7525" (UID: "43866555-48c4-49e6-a026-f0382fed7525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:01.746492 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.746469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33b4db7-471f-422c-9b86-68c27ccec8ba-kube-api-access-kdbv9" (OuterVolumeSpecName: "kube-api-access-kdbv9") pod "a33b4db7-471f-422c-9b86-68c27ccec8ba" (UID: "a33b4db7-471f-422c-9b86-68c27ccec8ba"). InnerVolumeSpecName "kube-api-access-kdbv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:01.746576 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.746499 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43866555-48c4-49e6-a026-f0382fed7525-kube-api-access-nwhqf" (OuterVolumeSpecName: "kube-api-access-nwhqf") pod "43866555-48c4-49e6-a026-f0382fed7525" (UID: "43866555-48c4-49e6-a026-f0382fed7525"). InnerVolumeSpecName "kube-api-access-nwhqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:01.845408 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.845374 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhqf\" (UniqueName: \"kubernetes.io/projected/43866555-48c4-49e6-a026-f0382fed7525-kube-api-access-nwhqf\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:01.845408 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.845402 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33b4db7-471f-422c-9b86-68c27ccec8ba-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:01.845408 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.845411 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbv9\" (UniqueName: \"kubernetes.io/projected/a33b4db7-471f-422c-9b86-68c27ccec8ba-kube-api-access-kdbv9\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:01.845648 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:01.845421 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43866555-48c4-49e6-a026-f0382fed7525-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:02.015852 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.015815 2567 generic.go:356] "Generic (PLEG): container finished" podID="43866555-48c4-49e6-a026-f0382fed7525" containerID="ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40" exitCode=0 Aug 05 16:56:02.015852 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.015854 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" Aug 05 16:56:02.016110 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.015903 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" event={"ID":"43866555-48c4-49e6-a026-f0382fed7525","Type":"ContainerDied","Data":"ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40"} Aug 05 16:56:02.016110 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.015955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj" event={"ID":"43866555-48c4-49e6-a026-f0382fed7525","Type":"ContainerDied","Data":"10e3866d59f454122a106b3fe63f0618594f232c5b832ceb1d22f64d7ca20670"} Aug 05 16:56:02.016110 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.015978 2567 scope.go:117] "RemoveContainer" containerID="ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40" Aug 05 16:56:02.017400 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.017376 2567 generic.go:356] "Generic (PLEG): container finished" podID="a33b4db7-471f-422c-9b86-68c27ccec8ba" containerID="b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1" exitCode=0 Aug 05 16:56:02.017505 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.017417 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" Aug 05 16:56:02.017505 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.017474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" event={"ID":"a33b4db7-471f-422c-9b86-68c27ccec8ba","Type":"ContainerDied","Data":"b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1"} Aug 05 16:56:02.017591 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.017508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r" event={"ID":"a33b4db7-471f-422c-9b86-68c27ccec8ba","Type":"ContainerDied","Data":"413607b5923a77ae75c6987934b519ca70bbc31dca5925fe4852c14cff7fd2d2"} Aug 05 16:56:02.019160 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.019079 2567 generic.go:356] "Generic (PLEG): container finished" podID="c347e886-9247-42dc-a4b7-ffb5340b6574" containerID="76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09" exitCode=0 Aug 05 16:56:02.019160 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.019122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" event={"ID":"c347e886-9247-42dc-a4b7-ffb5340b6574","Type":"ContainerDied","Data":"76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09"} Aug 05 16:56:02.019160 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.019136 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" Aug 05 16:56:02.019370 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.019175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44" event={"ID":"c347e886-9247-42dc-a4b7-ffb5340b6574","Type":"ContainerDied","Data":"739f9b1d4a7e158971bae1e6b57b408d893234f08b528b7e58c3ec3a00588f43"} Aug 05 16:56:02.025682 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.025665 2567 scope.go:117] "RemoveContainer" containerID="ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40" Aug 05 16:56:02.025940 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:56:02.025917 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40\": container with ID starting with ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40 not found: ID does not exist" containerID="ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40" Aug 05 16:56:02.026045 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.025947 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40"} err="failed to get container status \"ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40\": rpc error: code = NotFound desc = could not find container \"ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40\": container with ID starting with ee78d4561e343e335c41d361132aa9f752066912669f8d26021cdef2bd5e3c40 not found: ID does not exist" Aug 05 16:56:02.026045 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.025971 2567 scope.go:117] "RemoveContainer" containerID="b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1" Aug 05 16:56:02.034164 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.034147 2567 scope.go:117] "RemoveContainer" containerID="b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1" Aug 05 16:56:02.034440 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:56:02.034412 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1\": container with ID starting with b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1 not found: ID does not exist" containerID="b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1" Aug 05 16:56:02.034540 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.034447 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1"} err="failed to get container status \"b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1\": rpc error: code = NotFound desc = could not find container \"b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1\": container with ID starting with b064f479a234e8f615b3d0ecc4d857c0813f0b3ea23ed6b0403877e74a1455f1 not found: ID does not exist" Aug 05 16:56:02.034540 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.034466 2567 scope.go:117] "RemoveContainer" containerID="76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09" Aug 05 16:56:02.042622 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.042590 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj"] Aug 05 16:56:02.043487 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.043471 2567 scope.go:117] "RemoveContainer" containerID="76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09" Aug 05 16:56:02.043765 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:56:02.043747 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09\": container with ID starting with 76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09 not found: ID does not exist" containerID="76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09" Aug 05 16:56:02.043808 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.043773 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09"} err="failed to get container status \"76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09\": rpc error: code = NotFound desc = could not find container \"76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09\": container with ID starting with 76369f69b796456af48b53091b138b0cd336dab9d7d854211b486bd2d5e3bf09 not found: ID does not exist" Aug 05 16:56:02.052013 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.051987 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a6cf6a5-dd4fbc666-686lj"] Aug 05 16:56:02.066527 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.066492 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r"] Aug 05 16:56:02.068956 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.068930 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a6cf6a5-846c98b96-cdm8r"] Aug 05 16:56:02.078159 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.078135 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44"] Aug 05 16:56:02.083727 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:02.083703 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3a6cf6a5-7b56ffb47c-qbm44"] Aug 05 16:56:03.011060 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:03.011019 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43866555-48c4-49e6-a026-f0382fed7525" path="/var/lib/kubelet/pods/43866555-48c4-49e6-a026-f0382fed7525/volumes" Aug 05 16:56:03.011728 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:03.011705 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33b4db7-471f-422c-9b86-68c27ccec8ba" path="/var/lib/kubelet/pods/a33b4db7-471f-422c-9b86-68c27ccec8ba/volumes" Aug 05 16:56:03.012177 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:03.012158 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c347e886-9247-42dc-a4b7-ffb5340b6574" path="/var/lib/kubelet/pods/c347e886-9247-42dc-a4b7-ffb5340b6574/volumes" Aug 05 16:56:06.880888 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.880855 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4"] Aug 05 16:56:06.881284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881202 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43866555-48c4-49e6-a026-f0382fed7525" containerName="payload-processing" Aug 05 16:56:06.881284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881214 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="43866555-48c4-49e6-a026-f0382fed7525" containerName="payload-processing" Aug 05 16:56:06.881284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881227 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33b4db7-471f-422c-9b86-68c27ccec8ba" containerName="payload-pre-processing" Aug 05 16:56:06.881284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881233 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33b4db7-471f-422c-9b86-68c27ccec8ba" containerName="payload-pre-processing" Aug 05 16:56:06.881284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881248 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c347e886-9247-42dc-a4b7-ffb5340b6574" containerName="maas-api" Aug 05 16:56:06.881284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881258 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c347e886-9247-42dc-a4b7-ffb5340b6574" containerName="maas-api" Aug 05 16:56:06.881476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881308 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33b4db7-471f-422c-9b86-68c27ccec8ba" containerName="payload-pre-processing" Aug 05 16:56:06.881476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881315 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c347e886-9247-42dc-a4b7-ffb5340b6574" containerName="maas-api" Aug 05 16:56:06.881476 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.881322 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="43866555-48c4-49e6-a026-f0382fed7525" containerName="payload-processing" Aug 05 16:56:06.884872 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.884853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:06.891065 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.891032 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0ac50acb-serving-cert\"" Aug 05 16:56:06.891241 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.891060 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 16:56:06.891870 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.891847 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 16:56:06.892811 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.892791 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:56:06.908746 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.908722 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4"] Aug 05 16:56:06.991345 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.991313 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/decca532-cd72-4e60-851c-17c80d9cd307-maas-api-tls\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:06.991531 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.991374 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/decca532-cd72-4e60-851c-17c80d9cd307-openshift-service-ca\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:06.991531 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:06.991460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkmh\" (UniqueName: \"kubernetes.io/projected/decca532-cd72-4e60-851c-17c80d9cd307-kube-api-access-nxkmh\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.092368 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.092338 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/decca532-cd72-4e60-851c-17c80d9cd307-openshift-service-ca\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.092566 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.092376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkmh\" (UniqueName: \"kubernetes.io/projected/decca532-cd72-4e60-851c-17c80d9cd307-kube-api-access-nxkmh\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.092566 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.092519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/decca532-cd72-4e60-851c-17c80d9cd307-maas-api-tls\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.092977 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.092950 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/decca532-cd72-4e60-851c-17c80d9cd307-openshift-service-ca\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.094999 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.094979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/decca532-cd72-4e60-851c-17c80d9cd307-maas-api-tls\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.106354 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.106333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkmh\" (UniqueName: \"kubernetes.io/projected/decca532-cd72-4e60-851c-17c80d9cd307-kube-api-access-nxkmh\") pod \"maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.195448 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.195374 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:07.339376 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.339350 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4"] Aug 05 16:56:07.341248 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:56:07.341218 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecca532_cd72_4e60_851c_17c80d9cd307.slice/crio-9510e04f101596412ad5d5deca057cf1db7894c9fc850ba1846fc80fa2040627 WatchSource:0}: Error finding container 9510e04f101596412ad5d5deca057cf1db7894c9fc850ba1846fc80fa2040627: Status 404 returned error can't find the container with id 9510e04f101596412ad5d5deca057cf1db7894c9fc850ba1846fc80fa2040627 Aug 05 16:56:07.975748 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.975721 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl"] Aug 05 16:56:07.979979 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.979961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:07.982923 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.982903 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0ac50acb-dockercfg-54948\"" Aug 05 16:56:07.983013 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:07.982929 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0ac50acb\"" Aug 05 16:56:08.001229 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.001171 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl"] Aug 05 16:56:08.003962 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.003935 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt"] Aug 05 16:56:08.007734 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.007713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.026293 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.026226 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt"] Aug 05 16:56:08.055255 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.055227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" event={"ID":"decca532-cd72-4e60-851c-17c80d9cd307","Type":"ContainerStarted","Data":"c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993"} Aug 05 16:56:08.055394 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.055275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" event={"ID":"decca532-cd72-4e60-851c-17c80d9cd307","Type":"ContainerStarted","Data":"9510e04f101596412ad5d5deca057cf1db7894c9fc850ba1846fc80fa2040627"} Aug 05 16:56:08.055394 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.055307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:08.081536 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.081492 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" podStartSLOduration=1.470040715 podStartE2EDuration="2.081475924s" totalImagesPullingTime="611.435209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:56:08.080298183 +0000 UTC m=+1999.540100488" watchObservedRunningTime="2026-08-05 16:56:08.081475924 +0000 UTC m=+1999.541278205" Aug 05 16:56:08.100898 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.100864 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrnt\" (UniqueName: \"kubernetes.io/projected/5ffab0e3-6b02-423e-877f-f52fca19bc43-kube-api-access-ljrnt\") pod \"payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.101024 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.100928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eef376-94d9-4a5b-a196-46f1db092641-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.101076 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.101053 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7zs\" (UniqueName: \"kubernetes.io/projected/b5eef376-94d9-4a5b-a196-46f1db092641-kube-api-access-kf7zs\") pod \"payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.101133 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.101115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffab0e3-6b02-423e-877f-f52fca19bc43-plugins-config-volume\") pod \"payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.202196 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.202161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7zs\" (UniqueName: \"kubernetes.io/projected/b5eef376-94d9-4a5b-a196-46f1db092641-kube-api-access-kf7zs\") pod \"payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.202367 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.202213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffab0e3-6b02-423e-877f-f52fca19bc43-plugins-config-volume\") pod \"payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.202412 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.202377 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrnt\" (UniqueName: \"kubernetes.io/projected/5ffab0e3-6b02-423e-877f-f52fca19bc43-kube-api-access-ljrnt\") pod \"payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.202465 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.202452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eef376-94d9-4a5b-a196-46f1db092641-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.202995 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.202974 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffab0e3-6b02-423e-877f-f52fca19bc43-plugins-config-volume\") pod \"payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.203154 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.203137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eef376-94d9-4a5b-a196-46f1db092641-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.211372 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.211315 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrnt\" (UniqueName: \"kubernetes.io/projected/5ffab0e3-6b02-423e-877f-f52fca19bc43-kube-api-access-ljrnt\") pod \"payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.211461 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.211428 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7zs\" (UniqueName: \"kubernetes.io/projected/b5eef376-94d9-4a5b-a196-46f1db092641-kube-api-access-kf7zs\") pod \"payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.306519 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.306481 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:08.325667 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.325638 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:08.450494 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.450438 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl"] Aug 05 16:56:08.453259 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:56:08.453227 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffab0e3_6b02_423e_877f_f52fca19bc43.slice/crio-32a2b17339887ebfe44bccb3c84a69a5080d94a56c94952bf4f1f4c9befd9681 WatchSource:0}: Error finding container 32a2b17339887ebfe44bccb3c84a69a5080d94a56c94952bf4f1f4c9befd9681: Status 404 returned error can't find the container with id 32a2b17339887ebfe44bccb3c84a69a5080d94a56c94952bf4f1f4c9befd9681 Aug 05 16:56:08.484044 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:08.484016 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt"] Aug 05 16:56:08.485833 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:56:08.485810 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eef376_94d9_4a5b_a196_46f1db092641.slice/crio-8ee04f16929abde38dd1bcff791f397afd9aa50d9b8604757a1060a55c4a5bf5 WatchSource:0}: Error finding container 8ee04f16929abde38dd1bcff791f397afd9aa50d9b8604757a1060a55c4a5bf5: Status 404 returned error can't find the container with id 8ee04f16929abde38dd1bcff791f397afd9aa50d9b8604757a1060a55c4a5bf5 Aug 05 16:56:09.061542 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.061506 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" event={"ID":"b5eef376-94d9-4a5b-a196-46f1db092641","Type":"ContainerStarted","Data":"49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935"} Aug 05 16:56:09.062034 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.061656 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:09.062034 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.061700 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" event={"ID":"b5eef376-94d9-4a5b-a196-46f1db092641","Type":"ContainerStarted","Data":"8ee04f16929abde38dd1bcff791f397afd9aa50d9b8604757a1060a55c4a5bf5"} Aug 05 16:56:09.063037 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.063013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" event={"ID":"5ffab0e3-6b02-423e-877f-f52fca19bc43","Type":"ContainerStarted","Data":"8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612"} Aug 05 16:56:09.063161 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.063058 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" event={"ID":"5ffab0e3-6b02-423e-877f-f52fca19bc43","Type":"ContainerStarted","Data":"32a2b17339887ebfe44bccb3c84a69a5080d94a56c94952bf4f1f4c9befd9681"} Aug 05 16:56:09.080633 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.080585 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" podStartSLOduration=2.080572231 podStartE2EDuration="2.080572231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:56:09.077819347 +0000 UTC m=+2000.537621633" watchObservedRunningTime="2026-08-05 16:56:09.080572231 +0000 UTC m=+2000.540374512" Aug 05 16:56:09.096984 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:09.096939 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" podStartSLOduration=2.096925331 podStartE2EDuration="2.096925331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:56:09.094848083 +0000 UTC m=+2000.554650363" watchObservedRunningTime="2026-08-05 16:56:09.096925331 +0000 UTC m=+2000.556727624" Aug 05 16:56:10.069041 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:10.069009 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:14.068905 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:14.068873 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:20.071508 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:20.071474 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:21.075564 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:21.075525 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:37.614947 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.614909 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4"] Aug 05 16:56:37.615392 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.615169 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" podUID="decca532-cd72-4e60-851c-17c80d9cd307" containerName="maas-api" containerID="cri-o://c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993" gracePeriod=30 Aug 05 16:56:37.687029 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.686992 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl"] Aug 05 16:56:37.687292 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.687240 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" podUID="5ffab0e3-6b02-423e-877f-f52fca19bc43" containerName="payload-processing" containerID="cri-o://8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612" gracePeriod=30 Aug 05 16:56:37.699857 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.699832 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt"] Aug 05 16:56:37.700057 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.700034 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" podUID="b5eef376-94d9-4a5b-a196-46f1db092641" containerName="payload-pre-processing" containerID="cri-o://49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935" gracePeriod=30 Aug 05 16:56:37.959651 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.959622 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:37.981139 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.981113 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:37.987629 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:37.987604 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:38.048033 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048004 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/decca532-cd72-4e60-851c-17c80d9cd307-openshift-service-ca\") pod \"decca532-cd72-4e60-851c-17c80d9cd307\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " Aug 05 16:56:38.048221 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048039 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxkmh\" (UniqueName: \"kubernetes.io/projected/decca532-cd72-4e60-851c-17c80d9cd307-kube-api-access-nxkmh\") pod \"decca532-cd72-4e60-851c-17c80d9cd307\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " Aug 05 16:56:38.048221 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048076 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eef376-94d9-4a5b-a196-46f1db092641-plugins-config-volume\") pod \"b5eef376-94d9-4a5b-a196-46f1db092641\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " Aug 05 16:56:38.048348 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048236 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/decca532-cd72-4e60-851c-17c80d9cd307-maas-api-tls\") pod \"decca532-cd72-4e60-851c-17c80d9cd307\" (UID: \"decca532-cd72-4e60-851c-17c80d9cd307\") " Aug 05 16:56:38.048407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048385 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljrnt\" (UniqueName: \"kubernetes.io/projected/5ffab0e3-6b02-423e-877f-f52fca19bc43-kube-api-access-ljrnt\") pod \"5ffab0e3-6b02-423e-877f-f52fca19bc43\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " Aug 05 16:56:38.048463 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048420 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf7zs\" (UniqueName: \"kubernetes.io/projected/b5eef376-94d9-4a5b-a196-46f1db092641-kube-api-access-kf7zs\") pod \"b5eef376-94d9-4a5b-a196-46f1db092641\" (UID: \"b5eef376-94d9-4a5b-a196-46f1db092641\") " Aug 05 16:56:38.048463 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048450 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffab0e3-6b02-423e-877f-f52fca19bc43-plugins-config-volume\") pod \"5ffab0e3-6b02-423e-877f-f52fca19bc43\" (UID: \"5ffab0e3-6b02-423e-877f-f52fca19bc43\") " Aug 05 16:56:38.048560 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048467 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decca532-cd72-4e60-851c-17c80d9cd307-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "decca532-cd72-4e60-851c-17c80d9cd307" (UID: "decca532-cd72-4e60-851c-17c80d9cd307"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:38.048560 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.048487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5eef376-94d9-4a5b-a196-46f1db092641-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5eef376-94d9-4a5b-a196-46f1db092641" (UID: "b5eef376-94d9-4a5b-a196-46f1db092641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:38.049109 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.049052 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffab0e3-6b02-423e-877f-f52fca19bc43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ffab0e3-6b02-423e-877f-f52fca19bc43" (UID: "5ffab0e3-6b02-423e-877f-f52fca19bc43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:38.049322 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.049300 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/decca532-cd72-4e60-851c-17c80d9cd307-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.049402 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.049331 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5eef376-94d9-4a5b-a196-46f1db092641-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.052849 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.052814 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decca532-cd72-4e60-851c-17c80d9cd307-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "decca532-cd72-4e60-851c-17c80d9cd307" (UID: "decca532-cd72-4e60-851c-17c80d9cd307"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:56:38.052849 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.052835 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffab0e3-6b02-423e-877f-f52fca19bc43-kube-api-access-ljrnt" (OuterVolumeSpecName: "kube-api-access-ljrnt") pod "5ffab0e3-6b02-423e-877f-f52fca19bc43" (UID: "5ffab0e3-6b02-423e-877f-f52fca19bc43"). InnerVolumeSpecName "kube-api-access-ljrnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:38.052986 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.052870 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decca532-cd72-4e60-851c-17c80d9cd307-kube-api-access-nxkmh" (OuterVolumeSpecName: "kube-api-access-nxkmh") pod "decca532-cd72-4e60-851c-17c80d9cd307" (UID: "decca532-cd72-4e60-851c-17c80d9cd307"). InnerVolumeSpecName "kube-api-access-nxkmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:38.055528 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.055501 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eef376-94d9-4a5b-a196-46f1db092641-kube-api-access-kf7zs" (OuterVolumeSpecName: "kube-api-access-kf7zs") pod "b5eef376-94d9-4a5b-a196-46f1db092641" (UID: "b5eef376-94d9-4a5b-a196-46f1db092641"). InnerVolumeSpecName "kube-api-access-kf7zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:38.150157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.150049 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrnt\" (UniqueName: \"kubernetes.io/projected/5ffab0e3-6b02-423e-877f-f52fca19bc43-kube-api-access-ljrnt\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.150157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.150078 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7zs\" (UniqueName: \"kubernetes.io/projected/b5eef376-94d9-4a5b-a196-46f1db092641-kube-api-access-kf7zs\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.150157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.150112 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ffab0e3-6b02-423e-877f-f52fca19bc43-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.150157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.150121 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkmh\" (UniqueName: \"kubernetes.io/projected/decca532-cd72-4e60-851c-17c80d9cd307-kube-api-access-nxkmh\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.150157 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.150130 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/decca532-cd72-4e60-851c-17c80d9cd307-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:56:38.213283 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.213248 2567 generic.go:356] "Generic (PLEG): container finished" podID="decca532-cd72-4e60-851c-17c80d9cd307" containerID="c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993" exitCode=0 Aug 05 16:56:38.213473 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.213294 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" Aug 05 16:56:38.213473 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.213344 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" event={"ID":"decca532-cd72-4e60-851c-17c80d9cd307","Type":"ContainerDied","Data":"c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993"} Aug 05 16:56:38.213473 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.213399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4" event={"ID":"decca532-cd72-4e60-851c-17c80d9cd307","Type":"ContainerDied","Data":"9510e04f101596412ad5d5deca057cf1db7894c9fc850ba1846fc80fa2040627"} Aug 05 16:56:38.213473 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.213429 2567 scope.go:117] "RemoveContainer" containerID="c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993" Aug 05 16:56:38.214702 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.214680 2567 generic.go:356] "Generic (PLEG): container finished" podID="5ffab0e3-6b02-423e-877f-f52fca19bc43" containerID="8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612" exitCode=0 Aug 05 16:56:38.214793 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.214725 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" Aug 05 16:56:38.214793 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.214768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" event={"ID":"5ffab0e3-6b02-423e-877f-f52fca19bc43","Type":"ContainerDied","Data":"8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612"} Aug 05 16:56:38.214892 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.214820 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl" event={"ID":"5ffab0e3-6b02-423e-877f-f52fca19bc43","Type":"ContainerDied","Data":"32a2b17339887ebfe44bccb3c84a69a5080d94a56c94952bf4f1f4c9befd9681"} Aug 05 16:56:38.216174 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.216151 2567 generic.go:356] "Generic (PLEG): container finished" podID="b5eef376-94d9-4a5b-a196-46f1db092641" containerID="49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935" exitCode=0 Aug 05 16:56:38.216291 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.216197 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" Aug 05 16:56:38.216291 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.216224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" event={"ID":"b5eef376-94d9-4a5b-a196-46f1db092641","Type":"ContainerDied","Data":"49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935"} Aug 05 16:56:38.216291 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.216265 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt" event={"ID":"b5eef376-94d9-4a5b-a196-46f1db092641","Type":"ContainerDied","Data":"8ee04f16929abde38dd1bcff791f397afd9aa50d9b8604757a1060a55c4a5bf5"} Aug 05 16:56:38.222847 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.222826 2567 scope.go:117] "RemoveContainer" containerID="c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993" Aug 05 16:56:38.223112 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:56:38.223078 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993\": container with ID starting with c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993 not found: ID does not exist" containerID="c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993" Aug 05 16:56:38.223199 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.223121 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993"} err="failed to get container status \"c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993\": rpc error: code = NotFound desc = could not find container \"c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993\": container with ID starting with c4a3a475163803184757e4fb1bf5482124da9aeda05b76ace89fef5ee3a67993 not found: ID does not exist" Aug 05 16:56:38.223199 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.223137 2567 scope.go:117] "RemoveContainer" containerID="8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612" Aug 05 16:56:38.231386 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.231370 2567 scope.go:117] "RemoveContainer" containerID="8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612" Aug 05 16:56:38.231636 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:56:38.231619 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612\": container with ID starting with 8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612 not found: ID does not exist" containerID="8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612" Aug 05 16:56:38.231678 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.231646 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612"} err="failed to get container status \"8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612\": rpc error: code = NotFound desc = could not find container \"8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612\": container with ID starting with 8b558aa30e3bb6c81e73f9612d8b024ae7e98e5031a47ea70cad902675151612 not found: ID does not exist" Aug 05 16:56:38.231678 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.231663 2567 scope.go:117] "RemoveContainer" containerID="49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935" Aug 05 16:56:38.238954 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.238927 2567 scope.go:117] "RemoveContainer" containerID="49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935" Aug 05 16:56:38.239235 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:56:38.239218 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935\": container with ID starting with 49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935 not found: ID does not exist" containerID="49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935" Aug 05 16:56:38.239324 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.239240 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935"} err="failed to get container status \"49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935\": rpc error: code = NotFound desc = could not find container \"49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935\": container with ID starting with 49f179361c6c96b6d5d62aa2dfeb46c29c536c1c7106c10cfbe2214e2e58c935 not found: ID does not exist" Aug 05 16:56:38.242424 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.242403 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl"] Aug 05 16:56:38.248627 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.248579 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0ac50acb-548cb7cbf8-mkvkl"] Aug 05 16:56:38.259077 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.259037 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4"] Aug 05 16:56:38.261178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.261157 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0ac50acb-bdcc4899b-dk7x4"] Aug 05 16:56:38.270897 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.270876 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt"] Aug 05 16:56:38.275898 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:38.275875 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0ac50acb-88687d4-jnwbt"] Aug 05 16:56:39.015919 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:39.013387 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ffab0e3-6b02-423e-877f-f52fca19bc43" path="/var/lib/kubelet/pods/5ffab0e3-6b02-423e-877f-f52fca19bc43/volumes" Aug 05 16:56:39.015919 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:39.013933 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5eef376-94d9-4a5b-a196-46f1db092641" path="/var/lib/kubelet/pods/b5eef376-94d9-4a5b-a196-46f1db092641/volumes" Aug 05 16:56:39.015919 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:56:39.014485 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="decca532-cd72-4e60-851c-17c80d9cd307" path="/var/lib/kubelet/pods/decca532-cd72-4e60-851c-17c80d9cd307/volumes" Aug 05 16:57:49.072658 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:57:49.072628 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:57:49.079459 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:57:49.079439 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 16:58:45.414810 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.414712 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk"] Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415178 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="decca532-cd72-4e60-851c-17c80d9cd307" containerName="maas-api" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415197 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="decca532-cd72-4e60-851c-17c80d9cd307" containerName="maas-api" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415229 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ffab0e3-6b02-423e-877f-f52fca19bc43" containerName="payload-processing" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415238 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ffab0e3-6b02-423e-877f-f52fca19bc43" containerName="payload-processing" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415248 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5eef376-94d9-4a5b-a196-46f1db092641" containerName="payload-pre-processing" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415256 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5eef376-94d9-4a5b-a196-46f1db092641" containerName="payload-pre-processing" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415336 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ffab0e3-6b02-423e-877f-f52fca19bc43" containerName="payload-processing" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415346 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5eef376-94d9-4a5b-a196-46f1db092641" containerName="payload-pre-processing" Aug 05 16:58:45.415414 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.415358 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="decca532-cd72-4e60-851c-17c80d9cd307" containerName="maas-api" Aug 05 16:58:45.418325 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.418303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.420510 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.420486 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 16:58:45.420764 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.420745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-04a2be95-serving-cert\"" Aug 05 16:58:45.420869 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.420793 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 16:58:45.420869 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.420818 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:58:45.428486 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.428465 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk"] Aug 05 16:58:45.459981 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.459952 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.460131 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.459983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d47e30-ad35-440f-a3ba-5cc95f09d852-openshift-service-ca\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.460131 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.460005 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/c7d47e30-ad35-440f-a3ba-5cc95f09d852-kube-api-access-4thkj\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.560482 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.560446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.560482 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.560481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d47e30-ad35-440f-a3ba-5cc95f09d852-openshift-service-ca\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.560684 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.560501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/c7d47e30-ad35-440f-a3ba-5cc95f09d852-kube-api-access-4thkj\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.560684 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:58:45.560621 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-serving-cert: secret "maas-api-e2e-mt-04a2be95-serving-cert" not found Aug 05 16:58:45.560768 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:58:45.560699 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls podName:c7d47e30-ad35-440f-a3ba-5cc95f09d852 nodeName:}" failed. No retries permitted until 2026-08-05 16:58:46.060678393 +0000 UTC m=+2157.520480677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls") pod "maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" (UID: "c7d47e30-ad35-440f-a3ba-5cc95f09d852") : secret "maas-api-e2e-mt-04a2be95-serving-cert" not found Aug 05 16:58:45.561182 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.561161 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d47e30-ad35-440f-a3ba-5cc95f09d852-openshift-service-ca\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:45.573106 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:45.573057 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/c7d47e30-ad35-440f-a3ba-5cc95f09d852-kube-api-access-4thkj\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:46.064574 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.064538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:46.064755 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:58:46.064661 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-serving-cert: secret "maas-api-e2e-mt-04a2be95-serving-cert" not found Aug 05 16:58:46.064755 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:58:46.064721 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls podName:c7d47e30-ad35-440f-a3ba-5cc95f09d852 nodeName:}" failed. No retries permitted until 2026-08-05 16:58:47.064706627 +0000 UTC m=+2158.524508886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls") pod "maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" (UID: "c7d47e30-ad35-440f-a3ba-5cc95f09d852") : secret "maas-api-e2e-mt-04a2be95-serving-cert" not found Aug 05 16:58:46.098649 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.098616 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9"] Aug 05 16:58:46.101879 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.101855 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.104816 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.104796 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04a2be95\"" Aug 05 16:58:46.104940 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.104825 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04a2be95-dockercfg-rjzhg\"" Aug 05 16:58:46.114893 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.114868 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9"] Aug 05 16:58:46.165688 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.165650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqv4\" (UniqueName: \"kubernetes.io/projected/b72d2949-a71c-47aa-bbab-aa3efcbe981e-kube-api-access-mnqv4\") pod \"payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.166284 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.166253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d2949-a71c-47aa-bbab-aa3efcbe981e-plugins-config-volume\") pod \"payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.168308 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.168286 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw"] Aug 05 16:58:46.171436 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.171421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.183948 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.183921 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw"] Aug 05 16:58:46.266830 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.266797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d2949-a71c-47aa-bbab-aa3efcbe981e-plugins-config-volume\") pod \"payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.266830 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.266834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750c445-2063-4b2f-8f31-9f50817ba33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.267028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.266893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmr5\" (UniqueName: \"kubernetes.io/projected/5750c445-2063-4b2f-8f31-9f50817ba33a-kube-api-access-txmr5\") pod \"payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.267028 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.266986 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqv4\" (UniqueName: \"kubernetes.io/projected/b72d2949-a71c-47aa-bbab-aa3efcbe981e-kube-api-access-mnqv4\") pod \"payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.267474 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.267457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d2949-a71c-47aa-bbab-aa3efcbe981e-plugins-config-volume\") pod \"payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.275673 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.275650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqv4\" (UniqueName: \"kubernetes.io/projected/b72d2949-a71c-47aa-bbab-aa3efcbe981e-kube-api-access-mnqv4\") pod \"payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.368345 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.368310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750c445-2063-4b2f-8f31-9f50817ba33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.368534 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.368396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmr5\" (UniqueName: \"kubernetes.io/projected/5750c445-2063-4b2f-8f31-9f50817ba33a-kube-api-access-txmr5\") pod \"payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.369178 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.369157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750c445-2063-4b2f-8f31-9f50817ba33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.382471 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.382444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmr5\" (UniqueName: \"kubernetes.io/projected/5750c445-2063-4b2f-8f31-9f50817ba33a-kube-api-access-txmr5\") pod \"payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.412647 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.412617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.482057 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.482026 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.564128 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.564100 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9"] Aug 05 16:58:46.567273 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:58:46.567241 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72d2949_a71c_47aa_bbab_aa3efcbe981e.slice/crio-bdb948808a1c1e8eb9566b36f71e1141ff03275075c9ae82cb9c233b2179d11b WatchSource:0}: Error finding container bdb948808a1c1e8eb9566b36f71e1141ff03275075c9ae82cb9c233b2179d11b: Status 404 returned error can't find the container with id bdb948808a1c1e8eb9566b36f71e1141ff03275075c9ae82cb9c233b2179d11b Aug 05 16:58:46.619186 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.619128 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw"] Aug 05 16:58:46.622801 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:58:46.622776 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5750c445_2063_4b2f_8f31_9f50817ba33a.slice/crio-108d1ae3a13ca79a4414c12209db310d5e212cd1e53478871eddd703f74ff4dd WatchSource:0}: Error finding container 108d1ae3a13ca79a4414c12209db310d5e212cd1e53478871eddd703f74ff4dd: Status 404 returned error can't find the container with id 108d1ae3a13ca79a4414c12209db310d5e212cd1e53478871eddd703f74ff4dd Aug 05 16:58:46.825804 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.825764 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" event={"ID":"5750c445-2063-4b2f-8f31-9f50817ba33a","Type":"ContainerStarted","Data":"f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b"} Aug 05 16:58:46.826003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.825816 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" event={"ID":"5750c445-2063-4b2f-8f31-9f50817ba33a","Type":"ContainerStarted","Data":"108d1ae3a13ca79a4414c12209db310d5e212cd1e53478871eddd703f74ff4dd"} Aug 05 16:58:46.826003 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.825841 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:46.827351 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.827323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" event={"ID":"b72d2949-a71c-47aa-bbab-aa3efcbe981e","Type":"ContainerStarted","Data":"043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106"} Aug 05 16:58:46.827456 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.827373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" event={"ID":"b72d2949-a71c-47aa-bbab-aa3efcbe981e","Type":"ContainerStarted","Data":"bdb948808a1c1e8eb9566b36f71e1141ff03275075c9ae82cb9c233b2179d11b"} Aug 05 16:58:46.827456 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.827399 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:58:46.851510 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.851455 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" podStartSLOduration=0.85143726 podStartE2EDuration="851.43726ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:58:46.849243196 +0000 UTC m=+2158.309045480" watchObservedRunningTime="2026-08-05 16:58:46.85143726 +0000 UTC m=+2158.311239540" Aug 05 16:58:46.871363 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:46.871248 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" podStartSLOduration=0.871231222 podStartE2EDuration="871.231222ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:58:46.86923664 +0000 UTC m=+2158.329038943" watchObservedRunningTime="2026-08-05 16:58:46.871231222 +0000 UTC m=+2158.331033502" Aug 05 16:58:47.074935 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:47.074894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:47.077690 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:47.077663 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") pod \"maas-api-e2e-mt-04a2be95-5fd6645647-xlggk\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:47.230373 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:47.230281 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:47.370351 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:47.370323 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk"] Aug 05 16:58:47.373488 ip-10-0-131-60 kubenswrapper[2567]: W0805 16:58:47.373448 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d47e30_ad35_440f_a3ba_5cc95f09d852.slice/crio-caa86655e9303c9b6cd6883c8ea7ad921c14e9896aae1a7bf0e8a36c260ac06a WatchSource:0}: Error finding container caa86655e9303c9b6cd6883c8ea7ad921c14e9896aae1a7bf0e8a36c260ac06a: Status 404 returned error can't find the container with id caa86655e9303c9b6cd6883c8ea7ad921c14e9896aae1a7bf0e8a36c260ac06a Aug 05 16:58:47.833135 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:47.833101 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" event={"ID":"c7d47e30-ad35-440f-a3ba-5cc95f09d852","Type":"ContainerStarted","Data":"caa86655e9303c9b6cd6883c8ea7ad921c14e9896aae1a7bf0e8a36c260ac06a"} Aug 05 16:58:48.840436 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:48.840400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" event={"ID":"c7d47e30-ad35-440f-a3ba-5cc95f09d852","Type":"ContainerStarted","Data":"c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1"} Aug 05 16:58:48.840920 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:48.840479 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:48.858820 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:48.858766 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" podStartSLOduration=3.438541623 podStartE2EDuration="3.858750757s" totalImagesPullingTime="420.209134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:58:48.857388138 +0000 UTC m=+2160.317190420" watchObservedRunningTime="2026-08-05 16:58:48.858750757 +0000 UTC m=+2160.318553031" Aug 05 16:58:54.851598 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:54.851565 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:58:57.835587 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:57.835549 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:58:57.836036 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:58:57.835622 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:59:27.572407 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.572363 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk"] Aug 05 16:59:27.572948 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.572625 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" podUID="c7d47e30-ad35-440f-a3ba-5cc95f09d852" containerName="maas-api" containerID="cri-o://c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1" gracePeriod=30 Aug 05 16:59:27.653693 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.653657 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9"] Aug 05 16:59:27.653984 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.653958 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" podUID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" containerName="payload-processing" containerID="cri-o://043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106" gracePeriod=30 Aug 05 16:59:27.661921 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.661894 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw"] Aug 05 16:59:27.662173 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.662149 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" podUID="5750c445-2063-4b2f-8f31-9f50817ba33a" containerName="payload-pre-processing" containerID="cri-o://f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b" gracePeriod=30 Aug 05 16:59:27.834783 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.834512 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" podUID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.43:9004: connect: connection refused" Aug 05 16:59:27.834783 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.834635 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" podUID="5750c445-2063-4b2f-8f31-9f50817ba33a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.44:9004: connect: connection refused" Aug 05 16:59:27.911944 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.911914 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:59:27.927699 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.927671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d47e30-ad35-440f-a3ba-5cc95f09d852-openshift-service-ca\") pod \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " Aug 05 16:59:27.927904 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.927722 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") pod \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " Aug 05 16:59:27.927904 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.927818 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/c7d47e30-ad35-440f-a3ba-5cc95f09d852-kube-api-access-4thkj\") pod \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\" (UID: \"c7d47e30-ad35-440f-a3ba-5cc95f09d852\") " Aug 05 16:59:27.928426 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.928079 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d47e30-ad35-440f-a3ba-5cc95f09d852-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7d47e30-ad35-440f-a3ba-5cc95f09d852" (UID: "c7d47e30-ad35-440f-a3ba-5cc95f09d852"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:59:27.931251 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.931227 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d47e30-ad35-440f-a3ba-5cc95f09d852-kube-api-access-4thkj" (OuterVolumeSpecName: "kube-api-access-4thkj") pod "c7d47e30-ad35-440f-a3ba-5cc95f09d852" (UID: "c7d47e30-ad35-440f-a3ba-5cc95f09d852"). InnerVolumeSpecName "kube-api-access-4thkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:59:27.931357 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.931270 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7d47e30-ad35-440f-a3ba-5cc95f09d852" (UID: "c7d47e30-ad35-440f-a3ba-5cc95f09d852"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:59:27.943498 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.943473 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:59:27.951335 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:27.951315 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:59:28.028372 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028339 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnqv4\" (UniqueName: \"kubernetes.io/projected/b72d2949-a71c-47aa-bbab-aa3efcbe981e-kube-api-access-mnqv4\") pod \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " Aug 05 16:59:28.028372 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028375 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750c445-2063-4b2f-8f31-9f50817ba33a-plugins-config-volume\") pod \"5750c445-2063-4b2f-8f31-9f50817ba33a\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " Aug 05 16:59:28.028636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028419 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d2949-a71c-47aa-bbab-aa3efcbe981e-plugins-config-volume\") pod \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\" (UID: \"b72d2949-a71c-47aa-bbab-aa3efcbe981e\") " Aug 05 16:59:28.028636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028603 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/c7d47e30-ad35-440f-a3ba-5cc95f09d852-kube-api-access-4thkj\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.028636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028619 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d47e30-ad35-440f-a3ba-5cc95f09d852-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.028636 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028634 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d47e30-ad35-440f-a3ba-5cc95f09d852-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.028843 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028788 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5750c445-2063-4b2f-8f31-9f50817ba33a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5750c445-2063-4b2f-8f31-9f50817ba33a" (UID: "5750c445-2063-4b2f-8f31-9f50817ba33a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:59:28.028843 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.028818 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72d2949-a71c-47aa-bbab-aa3efcbe981e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b72d2949-a71c-47aa-bbab-aa3efcbe981e" (UID: "b72d2949-a71c-47aa-bbab-aa3efcbe981e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:59:28.029587 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.029559 2567 generic.go:356] "Generic (PLEG): container finished" podID="5750c445-2063-4b2f-8f31-9f50817ba33a" containerID="f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b" exitCode=0 Aug 05 16:59:28.029704 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.029621 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" Aug 05 16:59:28.029704 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.029641 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" event={"ID":"5750c445-2063-4b2f-8f31-9f50817ba33a","Type":"ContainerDied","Data":"f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b"} Aug 05 16:59:28.029821 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.029718 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw" event={"ID":"5750c445-2063-4b2f-8f31-9f50817ba33a","Type":"ContainerDied","Data":"108d1ae3a13ca79a4414c12209db310d5e212cd1e53478871eddd703f74ff4dd"} Aug 05 16:59:28.029821 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.029750 2567 scope.go:117] "RemoveContainer" containerID="f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b" Aug 05 16:59:28.030750 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.030687 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72d2949-a71c-47aa-bbab-aa3efcbe981e-kube-api-access-mnqv4" (OuterVolumeSpecName: "kube-api-access-mnqv4") pod "b72d2949-a71c-47aa-bbab-aa3efcbe981e" (UID: "b72d2949-a71c-47aa-bbab-aa3efcbe981e"). InnerVolumeSpecName "kube-api-access-mnqv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:59:28.031012 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.030996 2567 generic.go:356] "Generic (PLEG): container finished" podID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" containerID="043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106" exitCode=0 Aug 05 16:59:28.031080 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.031037 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" Aug 05 16:59:28.031080 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.031055 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" event={"ID":"b72d2949-a71c-47aa-bbab-aa3efcbe981e","Type":"ContainerDied","Data":"043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106"} Aug 05 16:59:28.031182 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.031115 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9" event={"ID":"b72d2949-a71c-47aa-bbab-aa3efcbe981e","Type":"ContainerDied","Data":"bdb948808a1c1e8eb9566b36f71e1141ff03275075c9ae82cb9c233b2179d11b"} Aug 05 16:59:28.032455 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.032431 2567 generic.go:356] "Generic (PLEG): container finished" podID="c7d47e30-ad35-440f-a3ba-5cc95f09d852" containerID="c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1" exitCode=0 Aug 05 16:59:28.032569 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.032479 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" Aug 05 16:59:28.032569 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.032492 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" event={"ID":"c7d47e30-ad35-440f-a3ba-5cc95f09d852","Type":"ContainerDied","Data":"c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1"} Aug 05 16:59:28.032569 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.032535 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk" event={"ID":"c7d47e30-ad35-440f-a3ba-5cc95f09d852","Type":"ContainerDied","Data":"caa86655e9303c9b6cd6883c8ea7ad921c14e9896aae1a7bf0e8a36c260ac06a"} Aug 05 16:59:28.039452 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.039425 2567 scope.go:117] "RemoveContainer" containerID="f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b" Aug 05 16:59:28.039708 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:59:28.039688 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b\": container with ID starting with f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b not found: ID does not exist" containerID="f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b" Aug 05 16:59:28.039788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.039717 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b"} err="failed to get container status \"f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b\": rpc error: code = NotFound desc = could not find container \"f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b\": container with ID starting with f4a02ea4b0eb6116b2d7b223693ad2105f765b4ce52810cc22384020e918c46b not found: ID does not exist" Aug 05 16:59:28.039788 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.039739 2567 scope.go:117] "RemoveContainer" containerID="043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106" Aug 05 16:59:28.047548 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.047532 2567 scope.go:117] "RemoveContainer" containerID="043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106" Aug 05 16:59:28.047802 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:59:28.047785 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106\": container with ID starting with 043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106 not found: ID does not exist" containerID="043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106" Aug 05 16:59:28.047847 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.047809 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106"} err="failed to get container status \"043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106\": rpc error: code = NotFound desc = could not find container \"043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106\": container with ID starting with 043c7c5bb9f123d54a09f9cfbfdab7de3a7859c386f0f2a73be711dbe5efd106 not found: ID does not exist" Aug 05 16:59:28.047847 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.047825 2567 scope.go:117] "RemoveContainer" containerID="c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1" Aug 05 16:59:28.054184 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.054151 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9"] Aug 05 16:59:28.055872 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.055857 2567 scope.go:117] "RemoveContainer" containerID="c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1" Aug 05 16:59:28.056142 ip-10-0-131-60 kubenswrapper[2567]: E0805 16:59:28.056124 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1\": container with ID starting with c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1 not found: ID does not exist" containerID="c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1" Aug 05 16:59:28.056194 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.056146 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1"} err="failed to get container status \"c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1\": rpc error: code = NotFound desc = could not find container \"c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1\": container with ID starting with c2e8979fb7082e1bd1cb2cc8705bd16505b573975c24086d62345014fecad6b1 not found: ID does not exist" Aug 05 16:59:28.062866 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.062845 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-04a2be95-7d9b5474-xq9h9"] Aug 05 16:59:28.073361 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.073307 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk"] Aug 05 16:59:28.078419 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.078396 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-04a2be95-5fd6645647-xlggk"] Aug 05 16:59:28.129277 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.129248 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txmr5\" (UniqueName: \"kubernetes.io/projected/5750c445-2063-4b2f-8f31-9f50817ba33a-kube-api-access-txmr5\") pod \"5750c445-2063-4b2f-8f31-9f50817ba33a\" (UID: \"5750c445-2063-4b2f-8f31-9f50817ba33a\") " Aug 05 16:59:28.129436 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.129424 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72d2949-a71c-47aa-bbab-aa3efcbe981e-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.129494 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.129438 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqv4\" (UniqueName: \"kubernetes.io/projected/b72d2949-a71c-47aa-bbab-aa3efcbe981e-kube-api-access-mnqv4\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.129494 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.129447 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750c445-2063-4b2f-8f31-9f50817ba33a-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.131314 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.131292 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5750c445-2063-4b2f-8f31-9f50817ba33a-kube-api-access-txmr5" (OuterVolumeSpecName: "kube-api-access-txmr5") pod "5750c445-2063-4b2f-8f31-9f50817ba33a" (UID: "5750c445-2063-4b2f-8f31-9f50817ba33a"). InnerVolumeSpecName "kube-api-access-txmr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:59:28.229988 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.229951 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txmr5\" (UniqueName: \"kubernetes.io/projected/5750c445-2063-4b2f-8f31-9f50817ba33a-kube-api-access-txmr5\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 16:59:28.351600 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.351567 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw"] Aug 05 16:59:28.358127 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:28.358074 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04a2be95-c8c585d88-l2ddw"] Aug 05 16:59:29.011063 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:29.011031 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5750c445-2063-4b2f-8f31-9f50817ba33a" path="/var/lib/kubelet/pods/5750c445-2063-4b2f-8f31-9f50817ba33a/volumes" Aug 05 16:59:29.011465 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:29.011405 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" path="/var/lib/kubelet/pods/b72d2949-a71c-47aa-bbab-aa3efcbe981e/volumes" Aug 05 16:59:29.011748 ip-10-0-131-60 kubenswrapper[2567]: I0805 16:59:29.011735 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d47e30-ad35-440f-a3ba-5cc95f09d852" path="/var/lib/kubelet/pods/c7d47e30-ad35-440f-a3ba-5cc95f09d852/volumes" Aug 05 17:01:56.394751 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.394717 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4"] Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395013 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5750c445-2063-4b2f-8f31-9f50817ba33a" containerName="payload-pre-processing" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395026 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5750c445-2063-4b2f-8f31-9f50817ba33a" containerName="payload-pre-processing" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395058 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" containerName="payload-processing" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395064 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" containerName="payload-processing" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395069 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d47e30-ad35-440f-a3ba-5cc95f09d852" containerName="maas-api" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395075 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d47e30-ad35-440f-a3ba-5cc95f09d852" containerName="maas-api" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395151 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72d2949-a71c-47aa-bbab-aa3efcbe981e" containerName="payload-processing" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395159 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d47e30-ad35-440f-a3ba-5cc95f09d852" containerName="maas-api" Aug 05 17:01:56.395188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.395166 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5750c445-2063-4b2f-8f31-9f50817ba33a" containerName="payload-pre-processing" Aug 05 17:01:56.397909 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.397889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.400191 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.400166 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-febb9564-serving-cert\"" Aug 05 17:01:56.400313 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.400168 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 17:01:56.400533 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.400510 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 17:01:56.400643 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.400549 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:01:56.410107 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.410068 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4"] Aug 05 17:01:56.458594 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.458554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6585c\" (UniqueName: \"kubernetes.io/projected/05bb6c49-adc2-49b8-a9cf-35519bb254f4-kube-api-access-6585c\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.458783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.458608 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05bb6c49-adc2-49b8-a9cf-35519bb254f4-maas-api-tls\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.458783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.458633 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb6c49-adc2-49b8-a9cf-35519bb254f4-openshift-service-ca\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.559309 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.559247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6585c\" (UniqueName: \"kubernetes.io/projected/05bb6c49-adc2-49b8-a9cf-35519bb254f4-kube-api-access-6585c\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.559513 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.559337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05bb6c49-adc2-49b8-a9cf-35519bb254f4-maas-api-tls\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.559513 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.559360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb6c49-adc2-49b8-a9cf-35519bb254f4-openshift-service-ca\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.559975 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.559951 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb6c49-adc2-49b8-a9cf-35519bb254f4-openshift-service-ca\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.561747 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.561728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05bb6c49-adc2-49b8-a9cf-35519bb254f4-maas-api-tls\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.567898 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.567869 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6585c\" (UniqueName: \"kubernetes.io/projected/05bb6c49-adc2-49b8-a9cf-35519bb254f4-kube-api-access-6585c\") pod \"maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.707902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.707805 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:56.846424 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.846391 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4"] Aug 05 17:01:56.850598 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:01:56.850566 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bb6c49_adc2_49b8_a9cf_35519bb254f4.slice/crio-e21e84d078daad54393d8e96db9bf0cfd8653f0822dcddeeb7a4fdd7b5a4c1cc WatchSource:0}: Error finding container e21e84d078daad54393d8e96db9bf0cfd8653f0822dcddeeb7a4fdd7b5a4c1cc: Status 404 returned error can't find the container with id e21e84d078daad54393d8e96db9bf0cfd8653f0822dcddeeb7a4fdd7b5a4c1cc Aug 05 17:01:56.851986 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:56.851966 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:01:57.186707 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.186670 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92"] Aug 05 17:01:57.191004 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.190973 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.193141 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.193119 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-febb9564\"" Aug 05 17:01:57.203522 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.203495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92"] Aug 05 17:01:57.211850 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.211822 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd"] Aug 05 17:01:57.215293 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.215269 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.217417 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.217398 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-febb9564-dockercfg-2fvgn\"" Aug 05 17:01:57.226456 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.226432 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd"] Aug 05 17:01:57.266867 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.266828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.266867 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.266868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8v5\" (UniqueName: \"kubernetes.io/projected/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-kube-api-access-4b8v5\") pod \"payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.368178 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.368149 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.368313 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.368189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ljpk\" (UniqueName: \"kubernetes.io/projected/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-kube-api-access-2ljpk\") pod \"payload-processing-e2e-mt-febb9564-85454df666-vjdkd\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.368496 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.368315 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8v5\" (UniqueName: \"kubernetes.io/projected/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-kube-api-access-4b8v5\") pod \"payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.368496 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.368356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-plugins-config-volume\") pod \"payload-processing-e2e-mt-febb9564-85454df666-vjdkd\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.369001 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.368976 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.377752 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.377733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8v5\" (UniqueName: \"kubernetes.io/projected/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-kube-api-access-4b8v5\") pod \"payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.468916 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.468819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ljpk\" (UniqueName: \"kubernetes.io/projected/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-kube-api-access-2ljpk\") pod \"payload-processing-e2e-mt-febb9564-85454df666-vjdkd\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.468916 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.468858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-plugins-config-volume\") pod \"payload-processing-e2e-mt-febb9564-85454df666-vjdkd\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.469605 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.469585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-plugins-config-volume\") pod \"payload-processing-e2e-mt-febb9564-85454df666-vjdkd\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.477241 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.477218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ljpk\" (UniqueName: \"kubernetes.io/projected/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-kube-api-access-2ljpk\") pod \"payload-processing-e2e-mt-febb9564-85454df666-vjdkd\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.502067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.502032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.527221 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.527189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.648949 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.648917 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92"] Aug 05 17:01:57.651015 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:01:57.650985 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ea909f_4abc_4b60_80b2_a7a62950a9b6.slice/crio-ff8e1245a8bedc21241328c58144855049d1fb03a22489f673c76c56953c95ca WatchSource:0}: Error finding container ff8e1245a8bedc21241328c58144855049d1fb03a22489f673c76c56953c95ca: Status 404 returned error can't find the container with id ff8e1245a8bedc21241328c58144855049d1fb03a22489f673c76c56953c95ca Aug 05 17:01:57.676832 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.676805 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd"] Aug 05 17:01:57.677593 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:01:57.677567 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9966c1_d2ce_4d8f_b0a0_17f51e91de98.slice/crio-26a7f42712ba3c8ce4bdc72ca1ccd29334d8e85789f5c70d54f371c17026e694 WatchSource:0}: Error finding container 26a7f42712ba3c8ce4bdc72ca1ccd29334d8e85789f5c70d54f371c17026e694: Status 404 returned error can't find the container with id 26a7f42712ba3c8ce4bdc72ca1ccd29334d8e85789f5c70d54f371c17026e694 Aug 05 17:01:57.744987 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.744952 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" event={"ID":"c5ea909f-4abc-4b60-80b2-a7a62950a9b6","Type":"ContainerStarted","Data":"7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe"} Aug 05 17:01:57.745160 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.745115 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" event={"ID":"c5ea909f-4abc-4b60-80b2-a7a62950a9b6","Type":"ContainerStarted","Data":"ff8e1245a8bedc21241328c58144855049d1fb03a22489f673c76c56953c95ca"} Aug 05 17:01:57.745160 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.745155 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:01:57.746576 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.746551 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" event={"ID":"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98","Type":"ContainerStarted","Data":"26a7f42712ba3c8ce4bdc72ca1ccd29334d8e85789f5c70d54f371c17026e694"} Aug 05 17:01:57.746693 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.746602 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:01:57.748230 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.748208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" event={"ID":"05bb6c49-adc2-49b8-a9cf-35519bb254f4","Type":"ContainerStarted","Data":"3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539"} Aug 05 17:01:57.748374 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.748253 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" event={"ID":"05bb6c49-adc2-49b8-a9cf-35519bb254f4","Type":"ContainerStarted","Data":"e21e84d078daad54393d8e96db9bf0cfd8653f0822dcddeeb7a4fdd7b5a4c1cc"} Aug 05 17:01:57.748374 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.748283 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:01:57.766579 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.766524 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" podStartSLOduration=0.766507608 podStartE2EDuration="766.507608ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:01:57.76449234 +0000 UTC m=+2349.224294622" watchObservedRunningTime="2026-08-05 17:01:57.766507608 +0000 UTC m=+2349.226309889" Aug 05 17:01:57.782754 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.782661 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" podStartSLOduration=0.782644393 podStartE2EDuration="782.644393ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:01:57.780027733 +0000 UTC m=+2349.239830014" watchObservedRunningTime="2026-08-05 17:01:57.782644393 +0000 UTC m=+2349.242446673" Aug 05 17:01:57.797594 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:57.797541 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" podStartSLOduration=1.30925191 podStartE2EDuration="1.797524828s" totalImagesPullingTime="488.272918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:01:57.796434298 +0000 UTC m=+2349.256236580" watchObservedRunningTime="2026-08-05 17:01:57.797524828 +0000 UTC m=+2349.257327120" Aug 05 17:01:58.755538 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:01:58.755491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" event={"ID":"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98","Type":"ContainerStarted","Data":"39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9"} Aug 05 17:02:03.760861 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:02:03.760832 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:02:08.757999 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:02:08.757970 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:02:09.763280 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:02:09.763251 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:02:49.109143 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:02:49.109110 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:02:49.113452 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:02:49.113428 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:03:16.413275 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.413187 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4"] Aug 05 17:03:16.413840 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.413407 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" podUID="05bb6c49-adc2-49b8-a9cf-35519bb254f4" containerName="maas-api" containerID="cri-o://3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539" gracePeriod=30 Aug 05 17:03:16.473018 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.472987 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd"] Aug 05 17:03:16.473262 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.473216 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" podUID="8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" containerName="payload-processing" containerID="cri-o://39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9" gracePeriod=30 Aug 05 17:03:16.481306 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.481115 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92"] Aug 05 17:03:16.481420 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.481324 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" podUID="c5ea909f-4abc-4b60-80b2-a7a62950a9b6" containerName="payload-pre-processing" containerID="cri-o://7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe" gracePeriod=30 Aug 05 17:03:16.746919 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.746895 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:03:16.764297 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.764270 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:03:16.775561 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.775539 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:03:16.845627 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845583 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-plugins-config-volume\") pod \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " Aug 05 17:03:16.845801 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845664 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6585c\" (UniqueName: \"kubernetes.io/projected/05bb6c49-adc2-49b8-a9cf-35519bb254f4-kube-api-access-6585c\") pod \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " Aug 05 17:03:16.845801 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845682 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b8v5\" (UniqueName: \"kubernetes.io/projected/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-kube-api-access-4b8v5\") pod \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\" (UID: \"c5ea909f-4abc-4b60-80b2-a7a62950a9b6\") " Aug 05 17:03:16.845801 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845700 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ljpk\" (UniqueName: \"kubernetes.io/projected/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-kube-api-access-2ljpk\") pod \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " Aug 05 17:03:16.845801 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845721 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb6c49-adc2-49b8-a9cf-35519bb254f4-openshift-service-ca\") pod \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " Aug 05 17:03:16.845801 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845789 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05bb6c49-adc2-49b8-a9cf-35519bb254f4-maas-api-tls\") pod \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\" (UID: \"05bb6c49-adc2-49b8-a9cf-35519bb254f4\") " Aug 05 17:03:16.846056 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.845808 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-plugins-config-volume\") pod \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\" (UID: \"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98\") " Aug 05 17:03:16.846222 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.846006 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5ea909f-4abc-4b60-80b2-a7a62950a9b6" (UID: "c5ea909f-4abc-4b60-80b2-a7a62950a9b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:03:16.846321 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.846231 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bb6c49-adc2-49b8-a9cf-35519bb254f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05bb6c49-adc2-49b8-a9cf-35519bb254f4" (UID: "05bb6c49-adc2-49b8-a9cf-35519bb254f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:03:16.846321 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.846312 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" (UID: "8a9966c1-d2ce-4d8f-b0a0-17f51e91de98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:03:16.847742 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.847717 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-kube-api-access-2ljpk" (OuterVolumeSpecName: "kube-api-access-2ljpk") pod "8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" (UID: "8a9966c1-d2ce-4d8f-b0a0-17f51e91de98"). InnerVolumeSpecName "kube-api-access-2ljpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:03:16.847865 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.847798 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05bb6c49-adc2-49b8-a9cf-35519bb254f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05bb6c49-adc2-49b8-a9cf-35519bb254f4" (UID: "05bb6c49-adc2-49b8-a9cf-35519bb254f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:03:16.848197 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.848172 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bb6c49-adc2-49b8-a9cf-35519bb254f4-kube-api-access-6585c" (OuterVolumeSpecName: "kube-api-access-6585c") pod "05bb6c49-adc2-49b8-a9cf-35519bb254f4" (UID: "05bb6c49-adc2-49b8-a9cf-35519bb254f4"). InnerVolumeSpecName "kube-api-access-6585c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:03:16.848197 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.848175 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-kube-api-access-4b8v5" (OuterVolumeSpecName: "kube-api-access-4b8v5") pod "c5ea909f-4abc-4b60-80b2-a7a62950a9b6" (UID: "c5ea909f-4abc-4b60-80b2-a7a62950a9b6"). InnerVolumeSpecName "kube-api-access-4b8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:03:16.946902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946841 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05bb6c49-adc2-49b8-a9cf-35519bb254f4-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:16.946902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946864 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:16.946902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946873 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:16.946902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946882 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6585c\" (UniqueName: \"kubernetes.io/projected/05bb6c49-adc2-49b8-a9cf-35519bb254f4-kube-api-access-6585c\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:16.946902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946890 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8v5\" (UniqueName: \"kubernetes.io/projected/c5ea909f-4abc-4b60-80b2-a7a62950a9b6-kube-api-access-4b8v5\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:16.946902 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946899 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ljpk\" (UniqueName: \"kubernetes.io/projected/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98-kube-api-access-2ljpk\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:16.947200 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:16.946907 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb6c49-adc2-49b8-a9cf-35519bb254f4-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:03:17.140646 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.140617 2567 generic.go:356] "Generic (PLEG): container finished" podID="c5ea909f-4abc-4b60-80b2-a7a62950a9b6" containerID="7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe" exitCode=0 Aug 05 17:03:17.140802 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.140667 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" Aug 05 17:03:17.140802 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.140706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" event={"ID":"c5ea909f-4abc-4b60-80b2-a7a62950a9b6","Type":"ContainerDied","Data":"7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe"} Aug 05 17:03:17.140802 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.140757 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92" event={"ID":"c5ea909f-4abc-4b60-80b2-a7a62950a9b6","Type":"ContainerDied","Data":"ff8e1245a8bedc21241328c58144855049d1fb03a22489f673c76c56953c95ca"} Aug 05 17:03:17.140802 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.140784 2567 scope.go:117] "RemoveContainer" containerID="7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe" Aug 05 17:03:17.142047 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.142026 2567 generic.go:356] "Generic (PLEG): container finished" podID="8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" containerID="39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9" exitCode=0 Aug 05 17:03:17.142170 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.142058 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" event={"ID":"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98","Type":"ContainerDied","Data":"39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9"} Aug 05 17:03:17.142170 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.142082 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" Aug 05 17:03:17.142170 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.142126 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd" event={"ID":"8a9966c1-d2ce-4d8f-b0a0-17f51e91de98","Type":"ContainerDied","Data":"26a7f42712ba3c8ce4bdc72ca1ccd29334d8e85789f5c70d54f371c17026e694"} Aug 05 17:03:17.143594 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.143562 2567 generic.go:356] "Generic (PLEG): container finished" podID="05bb6c49-adc2-49b8-a9cf-35519bb254f4" containerID="3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539" exitCode=0 Aug 05 17:03:17.143704 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.143611 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" Aug 05 17:03:17.143704 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.143616 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" event={"ID":"05bb6c49-adc2-49b8-a9cf-35519bb254f4","Type":"ContainerDied","Data":"3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539"} Aug 05 17:03:17.143704 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.143666 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4" event={"ID":"05bb6c49-adc2-49b8-a9cf-35519bb254f4","Type":"ContainerDied","Data":"e21e84d078daad54393d8e96db9bf0cfd8653f0822dcddeeb7a4fdd7b5a4c1cc"} Aug 05 17:03:17.150672 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.150655 2567 scope.go:117] "RemoveContainer" containerID="7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe" Aug 05 17:03:17.150937 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:03:17.150919 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe\": container with ID starting with 7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe not found: ID does not exist" containerID="7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe" Aug 05 17:03:17.150994 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.150948 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe"} err="failed to get container status \"7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe\": rpc error: code = NotFound desc = could not find container \"7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe\": container with ID starting with 7e96d67ec02bb9d42bfee8b52c047cafc6edaffda20b2e469a992624dcb5c7fe not found: ID does not exist" Aug 05 17:03:17.150994 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.150964 2567 scope.go:117] "RemoveContainer" containerID="39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9" Aug 05 17:03:17.158903 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.158878 2567 scope.go:117] "RemoveContainer" containerID="39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9" Aug 05 17:03:17.159398 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:03:17.159372 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9\": container with ID starting with 39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9 not found: ID does not exist" containerID="39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9" Aug 05 17:03:17.159486 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.159398 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9"} err="failed to get container status \"39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9\": rpc error: code = NotFound desc = could not find container \"39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9\": container with ID starting with 39326f3fae4e24a51c939c45fb858be6983c4be4214dc36e0649454999d62da9 not found: ID does not exist" Aug 05 17:03:17.159486 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.159414 2567 scope.go:117] "RemoveContainer" containerID="3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539" Aug 05 17:03:17.160986 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.160967 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd"] Aug 05 17:03:17.165066 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.165047 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-febb9564-85454df666-vjdkd"] Aug 05 17:03:17.167351 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.167333 2567 scope.go:117] "RemoveContainer" containerID="3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539" Aug 05 17:03:17.167584 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:03:17.167566 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539\": container with ID starting with 3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539 not found: ID does not exist" containerID="3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539" Aug 05 17:03:17.167633 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.167590 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539"} err="failed to get container status \"3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539\": rpc error: code = NotFound desc = could not find container \"3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539\": container with ID starting with 3e98b4cd36b40a439b13051761cfde4bc297ea39f219e2d2d45f9f87f782f539 not found: ID does not exist" Aug 05 17:03:17.174351 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.174327 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4"] Aug 05 17:03:17.176495 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.176475 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-febb9564-6c98ff8758-bs6r4"] Aug 05 17:03:17.186248 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.186229 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92"] Aug 05 17:03:17.190755 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:17.190730 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-febb9564-6f4777bfd8-5dn92"] Aug 05 17:03:19.015409 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:19.015379 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05bb6c49-adc2-49b8-a9cf-35519bb254f4" path="/var/lib/kubelet/pods/05bb6c49-adc2-49b8-a9cf-35519bb254f4/volumes" Aug 05 17:03:19.015798 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:19.015720 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" path="/var/lib/kubelet/pods/8a9966c1-d2ce-4d8f-b0a0-17f51e91de98/volumes" Aug 05 17:03:19.016021 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:03:19.016009 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5ea909f-4abc-4b60-80b2-a7a62950a9b6" path="/var/lib/kubelet/pods/c5ea909f-4abc-4b60-80b2-a7a62950a9b6/volumes" Aug 05 17:05:13.564734 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.564696 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq"] Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565028 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05bb6c49-adc2-49b8-a9cf-35519bb254f4" containerName="maas-api" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565040 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="05bb6c49-adc2-49b8-a9cf-35519bb254f4" containerName="maas-api" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565055 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" containerName="payload-processing" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565063 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" containerName="payload-processing" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565080 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ea909f-4abc-4b60-80b2-a7a62950a9b6" containerName="payload-pre-processing" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565100 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ea909f-4abc-4b60-80b2-a7a62950a9b6" containerName="payload-pre-processing" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565171 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5ea909f-4abc-4b60-80b2-a7a62950a9b6" containerName="payload-pre-processing" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565180 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="05bb6c49-adc2-49b8-a9cf-35519bb254f4" containerName="maas-api" Aug 05 17:05:13.565360 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.565185 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9966c1-d2ce-4d8f-b0a0-17f51e91de98" containerName="payload-processing" Aug 05 17:05:13.568209 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.568184 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.570371 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.570347 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:05:13.570495 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.570353 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 17:05:13.570729 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.570715 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 17:05:13.570831 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.570818 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-deb398-serving-cert\"" Aug 05 17:05:13.577474 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.577400 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq"] Aug 05 17:05:13.615696 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.615563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2888312-151b-4d92-9175-53ec396419b6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.615696 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.615608 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.615901 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.615717 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqv2\" (UniqueName: \"kubernetes.io/projected/e2888312-151b-4d92-9175-53ec396419b6-kube-api-access-jcqv2\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.717126 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.717074 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.717314 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.717154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqv2\" (UniqueName: \"kubernetes.io/projected/e2888312-151b-4d92-9175-53ec396419b6-kube-api-access-jcqv2\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.717314 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.717205 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2888312-151b-4d92-9175-53ec396419b6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.717314 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:05:13.717227 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-serving-cert: secret "maas-api-e2e-inf-a-deb398-serving-cert" not found Aug 05 17:05:13.717314 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:05:13.717299 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls podName:e2888312-151b-4d92-9175-53ec396419b6 nodeName:}" failed. No retries permitted until 2026-08-05 17:05:14.21728329 +0000 UTC m=+2545.677085548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls") pod "maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" (UID: "e2888312-151b-4d92-9175-53ec396419b6") : secret "maas-api-e2e-inf-a-deb398-serving-cert" not found Aug 05 17:05:13.717793 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.717776 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2888312-151b-4d92-9175-53ec396419b6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:13.725246 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:13.725220 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqv2\" (UniqueName: \"kubernetes.io/projected/e2888312-151b-4d92-9175-53ec396419b6-kube-api-access-jcqv2\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:14.222475 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.222424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:14.225153 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.225119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls\") pod \"maas-api-e2e-inf-a-deb398-75445c9764-kjpsq\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:14.318830 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.318773 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8"] Aug 05 17:05:14.322172 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.322154 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.324336 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.324311 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-deb398\"" Aug 05 17:05:14.324468 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.324314 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-deb398-dockercfg-r88d8\"" Aug 05 17:05:14.330059 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.330037 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2"] Aug 05 17:05:14.333606 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.333583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.334999 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.334976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8"] Aug 05 17:05:14.346441 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.346415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2"] Aug 05 17:05:14.424365 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.424329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2722debe-cff1-484e-a089-03f4dbe7a6fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.424365 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.424365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594d986a-5cf4-4370-b94b-a5457ffd43f8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.424631 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.424500 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grjl\" (UniqueName: \"kubernetes.io/projected/594d986a-5cf4-4370-b94b-a5457ffd43f8-kube-api-access-7grjl\") pod \"payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.424631 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.424556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7th\" (UniqueName: \"kubernetes.io/projected/2722debe-cff1-484e-a089-03f4dbe7a6fc-kube-api-access-bs7th\") pod \"payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.479817 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.479727 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:14.525920 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.525882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7th\" (UniqueName: \"kubernetes.io/projected/2722debe-cff1-484e-a089-03f4dbe7a6fc-kube-api-access-bs7th\") pod \"payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.526164 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.525947 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2722debe-cff1-484e-a089-03f4dbe7a6fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.526164 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.525965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594d986a-5cf4-4370-b94b-a5457ffd43f8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.526164 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.526003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grjl\" (UniqueName: \"kubernetes.io/projected/594d986a-5cf4-4370-b94b-a5457ffd43f8-kube-api-access-7grjl\") pod \"payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.526682 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.526660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2722debe-cff1-484e-a089-03f4dbe7a6fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.526749 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.526664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594d986a-5cf4-4370-b94b-a5457ffd43f8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.534521 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.534460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7th\" (UniqueName: \"kubernetes.io/projected/2722debe-cff1-484e-a089-03f4dbe7a6fc-kube-api-access-bs7th\") pod \"payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.535245 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.535222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grjl\" (UniqueName: \"kubernetes.io/projected/594d986a-5cf4-4370-b94b-a5457ffd43f8-kube-api-access-7grjl\") pod \"payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.616838 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.616812 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq"] Aug 05 17:05:14.618403 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:05:14.618380 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2888312_151b_4d92_9175_53ec396419b6.slice/crio-f7b6ff6f87e87a1a3063f88d15a5e2e32630c02eefe39497d8e6690a7b263d8b WatchSource:0}: Error finding container f7b6ff6f87e87a1a3063f88d15a5e2e32630c02eefe39497d8e6690a7b263d8b: Status 404 returned error can't find the container with id f7b6ff6f87e87a1a3063f88d15a5e2e32630c02eefe39497d8e6690a7b263d8b Aug 05 17:05:14.634214 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.634192 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:14.644368 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.644341 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:14.704183 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.704146 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" event={"ID":"e2888312-151b-4d92-9175-53ec396419b6","Type":"ContainerStarted","Data":"f7b6ff6f87e87a1a3063f88d15a5e2e32630c02eefe39497d8e6690a7b263d8b"} Aug 05 17:05:14.786169 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.786138 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8"] Aug 05 17:05:14.788584 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:05:14.788550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2722debe_cff1_484e_a089_03f4dbe7a6fc.slice/crio-db30437abae63d2ac2b80dcd6a924bfae85cc4b7529d35ff18431c0abb62c7df WatchSource:0}: Error finding container db30437abae63d2ac2b80dcd6a924bfae85cc4b7529d35ff18431c0abb62c7df: Status 404 returned error can't find the container with id db30437abae63d2ac2b80dcd6a924bfae85cc4b7529d35ff18431c0abb62c7df Aug 05 17:05:14.805503 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:14.805480 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2"] Aug 05 17:05:14.807250 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:05:14.807227 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594d986a_5cf4_4370_b94b_a5457ffd43f8.slice/crio-127e9c747f91fc23edecefc01a58ad600dc6f43d8ffa8de9ee16b134d01d1b6f WatchSource:0}: Error finding container 127e9c747f91fc23edecefc01a58ad600dc6f43d8ffa8de9ee16b134d01d1b6f: Status 404 returned error can't find the container with id 127e9c747f91fc23edecefc01a58ad600dc6f43d8ffa8de9ee16b134d01d1b6f Aug 05 17:05:15.710864 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.710830 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" event={"ID":"594d986a-5cf4-4370-b94b-a5457ffd43f8","Type":"ContainerStarted","Data":"3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd"} Aug 05 17:05:15.711353 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.710890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" event={"ID":"594d986a-5cf4-4370-b94b-a5457ffd43f8","Type":"ContainerStarted","Data":"127e9c747f91fc23edecefc01a58ad600dc6f43d8ffa8de9ee16b134d01d1b6f"} Aug 05 17:05:15.711353 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.711057 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:15.712452 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.712407 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" event={"ID":"2722debe-cff1-484e-a089-03f4dbe7a6fc","Type":"ContainerStarted","Data":"778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0"} Aug 05 17:05:15.712589 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.712472 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" event={"ID":"2722debe-cff1-484e-a089-03f4dbe7a6fc","Type":"ContainerStarted","Data":"db30437abae63d2ac2b80dcd6a924bfae85cc4b7529d35ff18431c0abb62c7df"} Aug 05 17:05:15.712589 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.712500 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:15.713939 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.713917 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" event={"ID":"e2888312-151b-4d92-9175-53ec396419b6","Type":"ContainerStarted","Data":"2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a"} Aug 05 17:05:15.714068 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.714051 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:15.727820 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.727765 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" podStartSLOduration=1.727746987 podStartE2EDuration="1.727746987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:15.725359424 +0000 UTC m=+2547.185161741" watchObservedRunningTime="2026-08-05 17:05:15.727746987 +0000 UTC m=+2547.187549269" Aug 05 17:05:15.742742 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:15.742696 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" podStartSLOduration=1.742678613 podStartE2EDuration="1.742678613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:15.741122882 +0000 UTC m=+2547.200925168" watchObservedRunningTime="2026-08-05 17:05:15.742678613 +0000 UTC m=+2547.202480894" Aug 05 17:05:21.727007 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:21.726978 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:05:21.743163 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:21.743107 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" podStartSLOduration=8.147807772 podStartE2EDuration="8.743070706s" totalImagesPullingTime="595.262934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:15.755619381 +0000 UTC m=+2547.215421662" watchObservedRunningTime="2026-08-05 17:05:21.743070706 +0000 UTC m=+2553.202872986" Aug 05 17:05:26.724201 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:26.724169 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:05:26.724594 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:26.724344 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:05:42.770638 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.770605 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k"] Aug 05 17:05:42.773827 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.773802 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.776401 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.776378 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3602fc-serving-cert\"" Aug 05 17:05:42.783583 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.783495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k"] Aug 05 17:05:42.886308 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.886274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.886476 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.886347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndh9\" (UniqueName: \"kubernetes.io/projected/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-kube-api-access-nndh9\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.886476 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.886394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-maas-api-tls\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.987362 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.987322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.987537 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.987383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndh9\" (UniqueName: \"kubernetes.io/projected/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-kube-api-access-nndh9\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.987537 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.987427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-maas-api-tls\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.987987 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.987963 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.989966 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.989937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-maas-api-tls\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:42.994836 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:42.994793 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndh9\" (UniqueName: \"kubernetes.io/projected/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-kube-api-access-nndh9\") pod \"maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:43.088431 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.088390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:43.225563 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.225535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k"] Aug 05 17:05:43.227376 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:05:43.227350 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f0ceec_ddc3_4385_bbe6_7cf045fe6962.slice/crio-257d5218e2545c99fa4d84414b3eee9bff24952f7d54c7bcc8091591a69dc725 WatchSource:0}: Error finding container 257d5218e2545c99fa4d84414b3eee9bff24952f7d54c7bcc8091591a69dc725: Status 404 returned error can't find the container with id 257d5218e2545c99fa4d84414b3eee9bff24952f7d54c7bcc8091591a69dc725 Aug 05 17:05:43.527807 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.527720 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h"] Aug 05 17:05:43.532131 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.532115 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.534635 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.534611 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3602fc-dockercfg-hfckb\"" Aug 05 17:05:43.534785 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.534621 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3602fc\"" Aug 05 17:05:43.541047 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.541026 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h"] Aug 05 17:05:43.591832 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.591792 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph"] Aug 05 17:05:43.595466 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.595442 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.609892 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.609867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph"] Aug 05 17:05:43.693504 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.693464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e987b667-37ef-48fd-bba5-fb89574fb774-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.693678 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.693525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2955d\" (UniqueName: \"kubernetes.io/projected/de84b677-13fc-4302-81cd-af12465b7af2-kube-api-access-2955d\") pod \"payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.693678 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.693638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfsf\" (UniqueName: \"kubernetes.io/projected/e987b667-37ef-48fd-bba5-fb89574fb774-kube-api-access-nzfsf\") pod \"payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.693756 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.693683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de84b677-13fc-4302-81cd-af12465b7af2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.794267 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.794191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfsf\" (UniqueName: \"kubernetes.io/projected/e987b667-37ef-48fd-bba5-fb89574fb774-kube-api-access-nzfsf\") pod \"payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.794267 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.794240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de84b677-13fc-4302-81cd-af12465b7af2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.794753 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.794362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e987b667-37ef-48fd-bba5-fb89574fb774-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.794753 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.794440 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2955d\" (UniqueName: \"kubernetes.io/projected/de84b677-13fc-4302-81cd-af12465b7af2-kube-api-access-2955d\") pod \"payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.794949 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.794932 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de84b677-13fc-4302-81cd-af12465b7af2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.795121 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.795079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e987b667-37ef-48fd-bba5-fb89574fb774-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.803010 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.802986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfsf\" (UniqueName: \"kubernetes.io/projected/e987b667-37ef-48fd-bba5-fb89574fb774-kube-api-access-nzfsf\") pod \"payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.803698 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.803679 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2955d\" (UniqueName: \"kubernetes.io/projected/de84b677-13fc-4302-81cd-af12465b7af2-kube-api-access-2955d\") pod \"payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.843746 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.843726 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:43.853401 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.853378 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" event={"ID":"77f0ceec-ddc3-4385-bbe6-7cf045fe6962","Type":"ContainerStarted","Data":"257d5218e2545c99fa4d84414b3eee9bff24952f7d54c7bcc8091591a69dc725"} Aug 05 17:05:43.908586 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.908549 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:43.993853 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:43.993824 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h"] Aug 05 17:05:43.994900 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:05:43.994855 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode987b667_37ef_48fd_bba5_fb89574fb774.slice/crio-eda464f44f140bfffbb6dad7587e0ca377e65a9f1c67222cb8e9f9e8166d92b4 WatchSource:0}: Error finding container eda464f44f140bfffbb6dad7587e0ca377e65a9f1c67222cb8e9f9e8166d92b4: Status 404 returned error can't find the container with id eda464f44f140bfffbb6dad7587e0ca377e65a9f1c67222cb8e9f9e8166d92b4 Aug 05 17:05:44.062297 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.062272 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph"] Aug 05 17:05:44.064450 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:05:44.064423 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde84b677_13fc_4302_81cd_af12465b7af2.slice/crio-a80dc03cebc52528c9ba6d1335d46560bc371edf8058890960b5ac5c65c28a0c WatchSource:0}: Error finding container a80dc03cebc52528c9ba6d1335d46560bc371edf8058890960b5ac5c65c28a0c: Status 404 returned error can't find the container with id a80dc03cebc52528c9ba6d1335d46560bc371edf8058890960b5ac5c65c28a0c Aug 05 17:05:44.860853 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.860818 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" event={"ID":"77f0ceec-ddc3-4385-bbe6-7cf045fe6962","Type":"ContainerStarted","Data":"9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f"} Aug 05 17:05:44.861332 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.860995 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:44.862404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.862379 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" event={"ID":"de84b677-13fc-4302-81cd-af12465b7af2","Type":"ContainerStarted","Data":"ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a"} Aug 05 17:05:44.862523 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.862432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" event={"ID":"de84b677-13fc-4302-81cd-af12465b7af2","Type":"ContainerStarted","Data":"a80dc03cebc52528c9ba6d1335d46560bc371edf8058890960b5ac5c65c28a0c"} Aug 05 17:05:44.862523 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.862462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:05:44.863889 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.863863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" event={"ID":"e987b667-37ef-48fd-bba5-fb89574fb774","Type":"ContainerStarted","Data":"bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22"} Aug 05 17:05:44.864018 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.863911 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" event={"ID":"e987b667-37ef-48fd-bba5-fb89574fb774","Type":"ContainerStarted","Data":"eda464f44f140bfffbb6dad7587e0ca377e65a9f1c67222cb8e9f9e8166d92b4"} Aug 05 17:05:44.864018 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.863937 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:44.879429 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.879390 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" podStartSLOduration=2.2881563209999998 podStartE2EDuration="2.879377366s" totalImagesPullingTime="591.221045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:44.876652425 +0000 UTC m=+2576.336454706" watchObservedRunningTime="2026-08-05 17:05:44.879377366 +0000 UTC m=+2576.339179646" Aug 05 17:05:44.892841 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.892802 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" podStartSLOduration=1.892789557 podStartE2EDuration="1.892789557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:44.891496457 +0000 UTC m=+2576.351298755" watchObservedRunningTime="2026-08-05 17:05:44.892789557 +0000 UTC m=+2576.352591837" Aug 05 17:05:44.906158 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:44.906122 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" podStartSLOduration=1.9061096530000001 podStartE2EDuration="1.906109653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:44.904947445 +0000 UTC m=+2576.364749725" watchObservedRunningTime="2026-08-05 17:05:44.906109653 +0000 UTC m=+2576.365911984" Aug 05 17:05:50.875488 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:50.875458 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:05:55.871884 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:55.871842 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:05:55.872371 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:05:55.872329 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:07:43.370997 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.370917 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq"] Aug 05 17:07:43.371571 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.371265 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" podUID="e2888312-151b-4d92-9175-53ec396419b6" containerName="maas-api" containerID="cri-o://2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a" gracePeriod=30 Aug 05 17:07:43.432102 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.432059 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2"] Aug 05 17:07:43.432332 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.432288 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" podUID="594d986a-5cf4-4370-b94b-a5457ffd43f8" containerName="payload-processing" containerID="cri-o://3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd" gracePeriod=30 Aug 05 17:07:43.449233 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.449200 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8"] Aug 05 17:07:43.449448 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.449412 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" podUID="2722debe-cff1-484e-a089-03f4dbe7a6fc" containerName="payload-pre-processing" containerID="cri-o://778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0" gracePeriod=30 Aug 05 17:07:43.700077 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.700006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:07:43.728216 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.728193 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:07:43.742559 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.742539 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:07:43.793151 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.793119 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls\") pod \"e2888312-151b-4d92-9175-53ec396419b6\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " Aug 05 17:07:43.793318 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.793159 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcqv2\" (UniqueName: \"kubernetes.io/projected/e2888312-151b-4d92-9175-53ec396419b6-kube-api-access-jcqv2\") pod \"e2888312-151b-4d92-9175-53ec396419b6\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " Aug 05 17:07:43.793318 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.793181 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2888312-151b-4d92-9175-53ec396419b6-openshift-service-ca\") pod \"e2888312-151b-4d92-9175-53ec396419b6\" (UID: \"e2888312-151b-4d92-9175-53ec396419b6\") " Aug 05 17:07:43.793606 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.793578 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2888312-151b-4d92-9175-53ec396419b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2888312-151b-4d92-9175-53ec396419b6" (UID: "e2888312-151b-4d92-9175-53ec396419b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:07:43.795304 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.795282 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2888312-151b-4d92-9175-53ec396419b6-kube-api-access-jcqv2" (OuterVolumeSpecName: "kube-api-access-jcqv2") pod "e2888312-151b-4d92-9175-53ec396419b6" (UID: "e2888312-151b-4d92-9175-53ec396419b6"). InnerVolumeSpecName "kube-api-access-jcqv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:43.795402 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.795381 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2888312-151b-4d92-9175-53ec396419b6" (UID: "e2888312-151b-4d92-9175-53ec396419b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:07:43.894533 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894470 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7grjl\" (UniqueName: \"kubernetes.io/projected/594d986a-5cf4-4370-b94b-a5457ffd43f8-kube-api-access-7grjl\") pod \"594d986a-5cf4-4370-b94b-a5457ffd43f8\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " Aug 05 17:07:43.894644 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894548 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2722debe-cff1-484e-a089-03f4dbe7a6fc-plugins-config-volume\") pod \"2722debe-cff1-484e-a089-03f4dbe7a6fc\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " Aug 05 17:07:43.894644 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894581 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs7th\" (UniqueName: \"kubernetes.io/projected/2722debe-cff1-484e-a089-03f4dbe7a6fc-kube-api-access-bs7th\") pod \"2722debe-cff1-484e-a089-03f4dbe7a6fc\" (UID: \"2722debe-cff1-484e-a089-03f4dbe7a6fc\") " Aug 05 17:07:43.894644 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894620 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594d986a-5cf4-4370-b94b-a5457ffd43f8-plugins-config-volume\") pod \"594d986a-5cf4-4370-b94b-a5457ffd43f8\" (UID: \"594d986a-5cf4-4370-b94b-a5457ffd43f8\") " Aug 05 17:07:43.894846 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894827 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2888312-151b-4d92-9175-53ec396419b6-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:43.894895 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894853 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2888312-151b-4d92-9175-53ec396419b6-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:43.894895 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894867 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqv2\" (UniqueName: \"kubernetes.io/projected/e2888312-151b-4d92-9175-53ec396419b6-kube-api-access-jcqv2\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:43.895005 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722debe-cff1-484e-a089-03f4dbe7a6fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2722debe-cff1-484e-a089-03f4dbe7a6fc" (UID: "2722debe-cff1-484e-a089-03f4dbe7a6fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:07:43.895060 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.894991 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594d986a-5cf4-4370-b94b-a5457ffd43f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "594d986a-5cf4-4370-b94b-a5457ffd43f8" (UID: "594d986a-5cf4-4370-b94b-a5457ffd43f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:07:43.896565 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.896544 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594d986a-5cf4-4370-b94b-a5457ffd43f8-kube-api-access-7grjl" (OuterVolumeSpecName: "kube-api-access-7grjl") pod "594d986a-5cf4-4370-b94b-a5457ffd43f8" (UID: "594d986a-5cf4-4370-b94b-a5457ffd43f8"). InnerVolumeSpecName "kube-api-access-7grjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:43.896565 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.896551 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2722debe-cff1-484e-a089-03f4dbe7a6fc-kube-api-access-bs7th" (OuterVolumeSpecName: "kube-api-access-bs7th") pod "2722debe-cff1-484e-a089-03f4dbe7a6fc" (UID: "2722debe-cff1-484e-a089-03f4dbe7a6fc"). InnerVolumeSpecName "kube-api-access-bs7th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:43.995980 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.995950 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7th\" (UniqueName: \"kubernetes.io/projected/2722debe-cff1-484e-a089-03f4dbe7a6fc-kube-api-access-bs7th\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:43.995980 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.995978 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594d986a-5cf4-4370-b94b-a5457ffd43f8-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:43.996198 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.995988 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grjl\" (UniqueName: \"kubernetes.io/projected/594d986a-5cf4-4370-b94b-a5457ffd43f8-kube-api-access-7grjl\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:43.996198 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:43.995998 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2722debe-cff1-484e-a089-03f4dbe7a6fc-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:07:44.460937 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.460900 2567 generic.go:356] "Generic (PLEG): container finished" podID="594d986a-5cf4-4370-b94b-a5457ffd43f8" containerID="3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd" exitCode=0 Aug 05 17:07:44.460937 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.460939 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" Aug 05 17:07:44.461566 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.460990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" event={"ID":"594d986a-5cf4-4370-b94b-a5457ffd43f8","Type":"ContainerDied","Data":"3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd"} Aug 05 17:07:44.461566 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.461045 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2" event={"ID":"594d986a-5cf4-4370-b94b-a5457ffd43f8","Type":"ContainerDied","Data":"127e9c747f91fc23edecefc01a58ad600dc6f43d8ffa8de9ee16b134d01d1b6f"} Aug 05 17:07:44.461566 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.461071 2567 scope.go:117] "RemoveContainer" containerID="3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd" Aug 05 17:07:44.462449 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.462425 2567 generic.go:356] "Generic (PLEG): container finished" podID="2722debe-cff1-484e-a089-03f4dbe7a6fc" containerID="778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0" exitCode=0 Aug 05 17:07:44.462534 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.462468 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" Aug 05 17:07:44.462534 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.462520 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" event={"ID":"2722debe-cff1-484e-a089-03f4dbe7a6fc","Type":"ContainerDied","Data":"778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0"} Aug 05 17:07:44.462622 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.462573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8" event={"ID":"2722debe-cff1-484e-a089-03f4dbe7a6fc","Type":"ContainerDied","Data":"db30437abae63d2ac2b80dcd6a924bfae85cc4b7529d35ff18431c0abb62c7df"} Aug 05 17:07:44.464126 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.464103 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2888312-151b-4d92-9175-53ec396419b6" containerID="2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a" exitCode=0 Aug 05 17:07:44.464238 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.464145 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" Aug 05 17:07:44.464238 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.464154 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" event={"ID":"e2888312-151b-4d92-9175-53ec396419b6","Type":"ContainerDied","Data":"2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a"} Aug 05 17:07:44.464238 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.464195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq" event={"ID":"e2888312-151b-4d92-9175-53ec396419b6","Type":"ContainerDied","Data":"f7b6ff6f87e87a1a3063f88d15a5e2e32630c02eefe39497d8e6690a7b263d8b"} Aug 05 17:07:44.471668 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.471649 2567 scope.go:117] "RemoveContainer" containerID="3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd" Aug 05 17:07:44.471994 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:07:44.471910 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd\": container with ID starting with 3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd not found: ID does not exist" containerID="3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd" Aug 05 17:07:44.471994 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.471934 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd"} err="failed to get container status \"3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd\": rpc error: code = NotFound desc = could not find container \"3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd\": container with ID starting with 3a11181412776904b6d533a3fb06812fa5896998d54c58ba0d8cff6bf758afcd not found: ID does not exist" Aug 05 17:07:44.471994 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.471950 2567 scope.go:117] "RemoveContainer" containerID="778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0" Aug 05 17:07:44.480648 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.480629 2567 scope.go:117] "RemoveContainer" containerID="778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0" Aug 05 17:07:44.480876 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:07:44.480859 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0\": container with ID starting with 778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0 not found: ID does not exist" containerID="778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0" Aug 05 17:07:44.480925 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.480883 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0"} err="failed to get container status \"778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0\": rpc error: code = NotFound desc = could not find container \"778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0\": container with ID starting with 778947eb9db54bd565a230560793072acb3496c5b82fce328207fdb123fbc6b0 not found: ID does not exist" Aug 05 17:07:44.480925 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.480899 2567 scope.go:117] "RemoveContainer" containerID="2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a" Aug 05 17:07:44.487296 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.487275 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2"] Aug 05 17:07:44.488466 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.488452 2567 scope.go:117] "RemoveContainer" containerID="2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a" Aug 05 17:07:44.488707 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:07:44.488691 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a\": container with ID starting with 2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a not found: ID does not exist" containerID="2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a" Aug 05 17:07:44.488746 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.488711 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a"} err="failed to get container status \"2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a\": rpc error: code = NotFound desc = could not find container \"2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a\": container with ID starting with 2afb00854caffc16b219f8d2f93b0d82ce50bee7c02c2caebfa1f3557a08400a not found: ID does not exist" Aug 05 17:07:44.491882 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.491861 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-deb398-547b5fd6b4-5jsv2"] Aug 05 17:07:44.501425 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.501399 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq"] Aug 05 17:07:44.507530 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.507489 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-deb398-75445c9764-kjpsq"] Aug 05 17:07:44.516726 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.516707 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8"] Aug 05 17:07:44.522786 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:44.522768 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-deb398-849f9ffc6-fz6h8"] Aug 05 17:07:45.016584 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:45.011790 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2722debe-cff1-484e-a089-03f4dbe7a6fc" path="/var/lib/kubelet/pods/2722debe-cff1-484e-a089-03f4dbe7a6fc/volumes" Aug 05 17:07:45.016584 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:45.012402 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594d986a-5cf4-4370-b94b-a5457ffd43f8" path="/var/lib/kubelet/pods/594d986a-5cf4-4370-b94b-a5457ffd43f8/volumes" Aug 05 17:07:45.016584 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:45.012932 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2888312-151b-4d92-9175-53ec396419b6" path="/var/lib/kubelet/pods/e2888312-151b-4d92-9175-53ec396419b6/volumes" Aug 05 17:07:49.139537 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:49.139511 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:07:49.145327 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:07:49.145306 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:08:00.915271 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:00.915229 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k"] Aug 05 17:08:00.915744 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:00.915493 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" podUID="77f0ceec-ddc3-4385-bbe6-7cf045fe6962" containerName="maas-api" containerID="cri-o://9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f" gracePeriod=30 Aug 05 17:08:00.977695 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:00.977659 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph"] Aug 05 17:08:00.977965 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:00.977933 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" podUID="de84b677-13fc-4302-81cd-af12465b7af2" containerName="payload-processing" containerID="cri-o://ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a" gracePeriod=30 Aug 05 17:08:00.983709 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:00.983480 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h"] Aug 05 17:08:00.984001 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:00.983957 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" podUID="e987b667-37ef-48fd-bba5-fb89574fb774" containerName="payload-pre-processing" containerID="cri-o://bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22" gracePeriod=30 Aug 05 17:08:01.243342 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.243317 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:08:01.270198 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.270168 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:08:01.277282 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.277260 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:08:01.344804 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344771 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nndh9\" (UniqueName: \"kubernetes.io/projected/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-kube-api-access-nndh9\") pod \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " Aug 05 17:08:01.344804 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e987b667-37ef-48fd-bba5-fb89574fb774-plugins-config-volume\") pod \"e987b667-37ef-48fd-bba5-fb89574fb774\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " Aug 05 17:08:01.345027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344822 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2955d\" (UniqueName: \"kubernetes.io/projected/de84b677-13fc-4302-81cd-af12465b7af2-kube-api-access-2955d\") pod \"de84b677-13fc-4302-81cd-af12465b7af2\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " Aug 05 17:08:01.345027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344842 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-maas-api-tls\") pod \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " Aug 05 17:08:01.345027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344863 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzfsf\" (UniqueName: \"kubernetes.io/projected/e987b667-37ef-48fd-bba5-fb89574fb774-kube-api-access-nzfsf\") pod \"e987b667-37ef-48fd-bba5-fb89574fb774\" (UID: \"e987b667-37ef-48fd-bba5-fb89574fb774\") " Aug 05 17:08:01.345027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344911 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de84b677-13fc-4302-81cd-af12465b7af2-plugins-config-volume\") pod \"de84b677-13fc-4302-81cd-af12465b7af2\" (UID: \"de84b677-13fc-4302-81cd-af12465b7af2\") " Aug 05 17:08:01.345027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.344962 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-openshift-service-ca\") pod \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\" (UID: \"77f0ceec-ddc3-4385-bbe6-7cf045fe6962\") " Aug 05 17:08:01.345315 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.345251 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e987b667-37ef-48fd-bba5-fb89574fb774-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e987b667-37ef-48fd-bba5-fb89574fb774" (UID: "e987b667-37ef-48fd-bba5-fb89574fb774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:08:01.345597 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.345574 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77f0ceec-ddc3-4385-bbe6-7cf045fe6962" (UID: "77f0ceec-ddc3-4385-bbe6-7cf045fe6962"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:08:01.345689 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.345584 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de84b677-13fc-4302-81cd-af12465b7af2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de84b677-13fc-4302-81cd-af12465b7af2" (UID: "de84b677-13fc-4302-81cd-af12465b7af2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:08:01.346970 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.346949 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-kube-api-access-nndh9" (OuterVolumeSpecName: "kube-api-access-nndh9") pod "77f0ceec-ddc3-4385-bbe6-7cf045fe6962" (UID: "77f0ceec-ddc3-4385-bbe6-7cf045fe6962"). InnerVolumeSpecName "kube-api-access-nndh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:08:01.347074 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.347023 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e987b667-37ef-48fd-bba5-fb89574fb774-kube-api-access-nzfsf" (OuterVolumeSpecName: "kube-api-access-nzfsf") pod "e987b667-37ef-48fd-bba5-fb89574fb774" (UID: "e987b667-37ef-48fd-bba5-fb89574fb774"). InnerVolumeSpecName "kube-api-access-nzfsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:08:01.347356 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.347332 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de84b677-13fc-4302-81cd-af12465b7af2-kube-api-access-2955d" (OuterVolumeSpecName: "kube-api-access-2955d") pod "de84b677-13fc-4302-81cd-af12465b7af2" (UID: "de84b677-13fc-4302-81cd-af12465b7af2"). InnerVolumeSpecName "kube-api-access-2955d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:08:01.347436 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.347332 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77f0ceec-ddc3-4385-bbe6-7cf045fe6962" (UID: "77f0ceec-ddc3-4385-bbe6-7cf045fe6962"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:08:01.445732 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445634 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndh9\" (UniqueName: \"kubernetes.io/projected/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-kube-api-access-nndh9\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.445732 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445667 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e987b667-37ef-48fd-bba5-fb89574fb774-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.445732 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445682 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2955d\" (UniqueName: \"kubernetes.io/projected/de84b677-13fc-4302-81cd-af12465b7af2-kube-api-access-2955d\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.445732 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445694 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.445732 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445706 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzfsf\" (UniqueName: \"kubernetes.io/projected/e987b667-37ef-48fd-bba5-fb89574fb774-kube-api-access-nzfsf\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.446012 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445742 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de84b677-13fc-4302-81cd-af12465b7af2-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.446012 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.445754 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f0ceec-ddc3-4385-bbe6-7cf045fe6962-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:08:01.548365 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.548330 2567 generic.go:356] "Generic (PLEG): container finished" podID="de84b677-13fc-4302-81cd-af12465b7af2" containerID="ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a" exitCode=0 Aug 05 17:08:01.548546 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.548376 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" Aug 05 17:08:01.548546 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.548407 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" event={"ID":"de84b677-13fc-4302-81cd-af12465b7af2","Type":"ContainerDied","Data":"ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a"} Aug 05 17:08:01.548546 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.548456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph" event={"ID":"de84b677-13fc-4302-81cd-af12465b7af2","Type":"ContainerDied","Data":"a80dc03cebc52528c9ba6d1335d46560bc371edf8058890960b5ac5c65c28a0c"} Aug 05 17:08:01.548546 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.548479 2567 scope.go:117] "RemoveContainer" containerID="ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a" Aug 05 17:08:01.549845 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.549822 2567 generic.go:356] "Generic (PLEG): container finished" podID="e987b667-37ef-48fd-bba5-fb89574fb774" containerID="bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22" exitCode=0 Aug 05 17:08:01.549953 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.549865 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" Aug 05 17:08:01.549953 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.549901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" event={"ID":"e987b667-37ef-48fd-bba5-fb89574fb774","Type":"ContainerDied","Data":"bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22"} Aug 05 17:08:01.549953 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.549949 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h" event={"ID":"e987b667-37ef-48fd-bba5-fb89574fb774","Type":"ContainerDied","Data":"eda464f44f140bfffbb6dad7587e0ca377e65a9f1c67222cb8e9f9e8166d92b4"} Aug 05 17:08:01.551326 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.551299 2567 generic.go:356] "Generic (PLEG): container finished" podID="77f0ceec-ddc3-4385-bbe6-7cf045fe6962" containerID="9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f" exitCode=0 Aug 05 17:08:01.551445 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.551328 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" event={"ID":"77f0ceec-ddc3-4385-bbe6-7cf045fe6962","Type":"ContainerDied","Data":"9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f"} Aug 05 17:08:01.551445 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.551347 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" Aug 05 17:08:01.551445 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.551375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k" event={"ID":"77f0ceec-ddc3-4385-bbe6-7cf045fe6962","Type":"ContainerDied","Data":"257d5218e2545c99fa4d84414b3eee9bff24952f7d54c7bcc8091591a69dc725"} Aug 05 17:08:01.559400 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.559374 2567 scope.go:117] "RemoveContainer" containerID="ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a" Aug 05 17:08:01.560348 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:08:01.559901 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a\": container with ID starting with ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a not found: ID does not exist" containerID="ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a" Aug 05 17:08:01.560348 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.560002 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a"} err="failed to get container status \"ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a\": rpc error: code = NotFound desc = could not find container \"ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a\": container with ID starting with ba8ca87d4703fc7106ec968d156322d5a7d3f922065c51b8a5bebeb614405f1a not found: ID does not exist" Aug 05 17:08:01.560348 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.560031 2567 scope.go:117] "RemoveContainer" containerID="bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22" Aug 05 17:08:01.572564 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.572543 2567 scope.go:117] "RemoveContainer" containerID="bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22" Aug 05 17:08:01.572871 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:08:01.572851 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22\": container with ID starting with bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22 not found: ID does not exist" containerID="bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22" Aug 05 17:08:01.572950 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.572885 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22"} err="failed to get container status \"bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22\": rpc error: code = NotFound desc = could not find container \"bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22\": container with ID starting with bc43427298cf502a8932fdc788a04acdc2da572a84ddc44d234898af5cd85b22 not found: ID does not exist" Aug 05 17:08:01.572950 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.572910 2567 scope.go:117] "RemoveContainer" containerID="9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f" Aug 05 17:08:01.580330 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.580300 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph"] Aug 05 17:08:01.582299 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.582283 2567 scope.go:117] "RemoveContainer" containerID="9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f" Aug 05 17:08:01.582599 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:08:01.582579 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f\": container with ID starting with 9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f not found: ID does not exist" containerID="9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f" Aug 05 17:08:01.582637 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.582606 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f"} err="failed to get container status \"9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f\": rpc error: code = NotFound desc = could not find container \"9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f\": container with ID starting with 9946b023af9b4c4f173360536382dad466f0d30b542717a45cf34f6d0b52075f not found: ID does not exist" Aug 05 17:08:01.585682 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.585655 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3602fc-76b5c8bcdd-lmkph"] Aug 05 17:08:01.595946 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.595913 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k"] Aug 05 17:08:01.599614 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.599592 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3602fc-5cd7b8974b-vgp4k"] Aug 05 17:08:01.609638 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.609613 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h"] Aug 05 17:08:01.614154 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:01.614132 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3602fc-6dc65b948d-xhb8h"] Aug 05 17:08:03.011181 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:03.011147 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f0ceec-ddc3-4385-bbe6-7cf045fe6962" path="/var/lib/kubelet/pods/77f0ceec-ddc3-4385-bbe6-7cf045fe6962/volumes" Aug 05 17:08:03.011563 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:03.011509 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de84b677-13fc-4302-81cd-af12465b7af2" path="/var/lib/kubelet/pods/de84b677-13fc-4302-81cd-af12465b7af2/volumes" Aug 05 17:08:03.011820 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:03.011807 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e987b667-37ef-48fd-bba5-fb89574fb774" path="/var/lib/kubelet/pods/e987b667-37ef-48fd-bba5-fb89574fb774/volumes" Aug 05 17:08:12.439783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.439744 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt"] Aug 05 17:08:12.440202 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440187 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594d986a-5cf4-4370-b94b-a5457ffd43f8" containerName="payload-processing" Aug 05 17:08:12.440253 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440205 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="594d986a-5cf4-4370-b94b-a5457ffd43f8" containerName="payload-processing" Aug 05 17:08:12.440253 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440222 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2888312-151b-4d92-9175-53ec396419b6" containerName="maas-api" Aug 05 17:08:12.440253 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440231 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2888312-151b-4d92-9175-53ec396419b6" containerName="maas-api" Aug 05 17:08:12.440253 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440241 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e987b667-37ef-48fd-bba5-fb89574fb774" containerName="payload-pre-processing" Aug 05 17:08:12.440253 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440250 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e987b667-37ef-48fd-bba5-fb89574fb774" containerName="payload-pre-processing" Aug 05 17:08:12.440404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440263 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f0ceec-ddc3-4385-bbe6-7cf045fe6962" containerName="maas-api" Aug 05 17:08:12.440404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440271 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f0ceec-ddc3-4385-bbe6-7cf045fe6962" containerName="maas-api" Aug 05 17:08:12.440404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440281 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de84b677-13fc-4302-81cd-af12465b7af2" containerName="payload-processing" Aug 05 17:08:12.440404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440289 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="de84b677-13fc-4302-81cd-af12465b7af2" containerName="payload-processing" Aug 05 17:08:12.440404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440325 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2722debe-cff1-484e-a089-03f4dbe7a6fc" containerName="payload-pre-processing" Aug 05 17:08:12.440404 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440334 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2722debe-cff1-484e-a089-03f4dbe7a6fc" containerName="payload-pre-processing" Aug 05 17:08:12.440582 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440406 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="de84b677-13fc-4302-81cd-af12465b7af2" containerName="payload-processing" Aug 05 17:08:12.440582 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440416 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="594d986a-5cf4-4370-b94b-a5457ffd43f8" containerName="payload-processing" Aug 05 17:08:12.440582 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440428 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2888312-151b-4d92-9175-53ec396419b6" containerName="maas-api" Aug 05 17:08:12.440582 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440437 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e987b667-37ef-48fd-bba5-fb89574fb774" containerName="payload-pre-processing" Aug 05 17:08:12.440582 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440446 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2722debe-cff1-484e-a089-03f4dbe7a6fc" containerName="payload-pre-processing" Aug 05 17:08:12.440582 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.440457 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f0ceec-ddc3-4385-bbe6-7cf045fe6962" containerName="maas-api" Aug 05 17:08:12.445252 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.445236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.447337 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.447311 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:08:12.448597 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.448575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2e153a-serving-cert\"" Aug 05 17:08:12.448896 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.448871 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 17:08:12.449003 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.448908 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 17:08:12.450590 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.450571 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt"] Aug 05 17:08:12.538783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.538747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db61b0-4243-4d22-95f3-845560dad361-maas-api-tls\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.538783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.538784 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgnnj\" (UniqueName: \"kubernetes.io/projected/c8db61b0-4243-4d22-95f3-845560dad361-kube-api-access-cgnnj\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.539011 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.538809 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db61b0-4243-4d22-95f3-845560dad361-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.640153 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.640120 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db61b0-4243-4d22-95f3-845560dad361-maas-api-tls\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.640153 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.640157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnnj\" (UniqueName: \"kubernetes.io/projected/c8db61b0-4243-4d22-95f3-845560dad361-kube-api-access-cgnnj\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.640416 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.640191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db61b0-4243-4d22-95f3-845560dad361-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.640925 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.640894 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db61b0-4243-4d22-95f3-845560dad361-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.642639 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.642618 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db61b0-4243-4d22-95f3-845560dad361-maas-api-tls\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.647514 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.647489 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnnj\" (UniqueName: \"kubernetes.io/projected/c8db61b0-4243-4d22-95f3-845560dad361-kube-api-access-cgnnj\") pod \"maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.758397 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.758308 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:12.891760 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.891735 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt"] Aug 05 17:08:12.894118 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:08:12.894069 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8db61b0_4243_4d22_95f3_845560dad361.slice/crio-3d35b6775b127897b128dd44431df7c79746e6a4b22e561f6887910b32f2f105 WatchSource:0}: Error finding container 3d35b6775b127897b128dd44431df7c79746e6a4b22e561f6887910b32f2f105: Status 404 returned error can't find the container with id 3d35b6775b127897b128dd44431df7c79746e6a4b22e561f6887910b32f2f105 Aug 05 17:08:12.895393 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:12.895370 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:08:13.110414 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.110379 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s"] Aug 05 17:08:13.114632 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.114607 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg"] Aug 05 17:08:13.114793 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.114724 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.116635 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.116613 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2e153a\"" Aug 05 17:08:13.116746 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.116658 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2e153a-dockercfg-zblr4\"" Aug 05 17:08:13.117761 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.117743 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.124430 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.124409 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s"] Aug 05 17:08:13.126661 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.126641 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg"] Aug 05 17:08:13.246774 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.246729 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0848a915-d13e-461c-80ba-3e5777461a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.246774 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.246776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkzd\" (UniqueName: \"kubernetes.io/projected/0848a915-d13e-461c-80ba-3e5777461a80-kube-api-access-8jkzd\") pod \"payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.247023 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.246836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c0328a-b509-4032-b874-98950a178324-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.247023 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.246890 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbdk\" (UniqueName: \"kubernetes.io/projected/d3c0328a-b509-4032-b874-98950a178324-kube-api-access-fxbdk\") pod \"payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.348120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.348067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkzd\" (UniqueName: \"kubernetes.io/projected/0848a915-d13e-461c-80ba-3e5777461a80-kube-api-access-8jkzd\") pod \"payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.348322 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.348287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c0328a-b509-4032-b874-98950a178324-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.348438 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.348348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbdk\" (UniqueName: \"kubernetes.io/projected/d3c0328a-b509-4032-b874-98950a178324-kube-api-access-fxbdk\") pod \"payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.348499 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.348455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0848a915-d13e-461c-80ba-3e5777461a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.349130 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.349074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c0328a-b509-4032-b874-98950a178324-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.349344 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.349317 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0848a915-d13e-461c-80ba-3e5777461a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.356927 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.356903 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkzd\" (UniqueName: \"kubernetes.io/projected/0848a915-d13e-461c-80ba-3e5777461a80-kube-api-access-8jkzd\") pod \"payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.357308 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.357285 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbdk\" (UniqueName: \"kubernetes.io/projected/d3c0328a-b509-4032-b874-98950a178324-kube-api-access-fxbdk\") pod \"payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.429419 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.429385 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:13.436418 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.436392 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:13.585995 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.585451 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s"] Aug 05 17:08:13.588374 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:08:13.588336 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0848a915_d13e_461c_80ba_3e5777461a80.slice/crio-ee4e0287b9c30831bffc9c08f12e030d0a76d4366db3d394431c76f626d9a8a4 WatchSource:0}: Error finding container ee4e0287b9c30831bffc9c08f12e030d0a76d4366db3d394431c76f626d9a8a4: Status 404 returned error can't find the container with id ee4e0287b9c30831bffc9c08f12e030d0a76d4366db3d394431c76f626d9a8a4 Aug 05 17:08:13.610966 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.610933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" event={"ID":"0848a915-d13e-461c-80ba-3e5777461a80","Type":"ContainerStarted","Data":"ee4e0287b9c30831bffc9c08f12e030d0a76d4366db3d394431c76f626d9a8a4"} Aug 05 17:08:13.612723 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.612700 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" event={"ID":"c8db61b0-4243-4d22-95f3-845560dad361","Type":"ContainerStarted","Data":"63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b"} Aug 05 17:08:13.612810 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.612745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" event={"ID":"c8db61b0-4243-4d22-95f3-845560dad361","Type":"ContainerStarted","Data":"3d35b6775b127897b128dd44431df7c79746e6a4b22e561f6887910b32f2f105"} Aug 05 17:08:13.612857 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.612831 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:13.616746 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.616726 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg"] Aug 05 17:08:13.619256 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:08:13.619231 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c0328a_b509_4032_b874_98950a178324.slice/crio-da1c84cbe59a76a5e688fd78cb3a417bd58063656309f60c4d5685abc0be8eb0 WatchSource:0}: Error finding container da1c84cbe59a76a5e688fd78cb3a417bd58063656309f60c4d5685abc0be8eb0: Status 404 returned error can't find the container with id da1c84cbe59a76a5e688fd78cb3a417bd58063656309f60c4d5685abc0be8eb0 Aug 05 17:08:13.631044 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:13.631002 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" podStartSLOduration=1.183659878 podStartE2EDuration="1.630986619s" totalImagesPullingTime="447.326741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:08:13.62983423 +0000 UTC m=+2725.089636512" watchObservedRunningTime="2026-08-05 17:08:13.630986619 +0000 UTC m=+2725.090788900" Aug 05 17:08:14.618731 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:14.618697 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" event={"ID":"0848a915-d13e-461c-80ba-3e5777461a80","Type":"ContainerStarted","Data":"4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e"} Aug 05 17:08:14.619286 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:14.618801 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:14.620243 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:14.620220 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" event={"ID":"d3c0328a-b509-4032-b874-98950a178324","Type":"ContainerStarted","Data":"25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d"} Aug 05 17:08:14.620327 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:14.620267 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" event={"ID":"d3c0328a-b509-4032-b874-98950a178324","Type":"ContainerStarted","Data":"da1c84cbe59a76a5e688fd78cb3a417bd58063656309f60c4d5685abc0be8eb0"} Aug 05 17:08:14.639527 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:14.639476 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" podStartSLOduration=1.639456382 podStartE2EDuration="1.639456382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:08:14.639321736 +0000 UTC m=+2726.099124022" watchObservedRunningTime="2026-08-05 17:08:14.639456382 +0000 UTC m=+2726.099258666" Aug 05 17:08:14.664051 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:14.664008 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" podStartSLOduration=1.663991442 podStartE2EDuration="1.663991442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:08:14.662646012 +0000 UTC m=+2726.122448297" watchObservedRunningTime="2026-08-05 17:08:14.663991442 +0000 UTC m=+2726.123793723" Aug 05 17:08:15.625220 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:15.625192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:19.625897 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:19.625863 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:08:25.627290 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:25.627256 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:08:26.631805 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:26.631773 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:08:41.620540 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.620503 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw"] Aug 05 17:08:41.624159 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.624137 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.626067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.626042 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2eaba9-serving-cert\"" Aug 05 17:08:41.636255 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.636225 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw"] Aug 05 17:08:41.688627 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.688594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhfb\" (UniqueName: \"kubernetes.io/projected/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-kube-api-access-mnhfb\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.688627 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.688630 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.688840 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.688716 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.789525 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.789481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhfb\" (UniqueName: \"kubernetes.io/projected/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-kube-api-access-mnhfb\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.789525 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.789525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.789734 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.789560 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.789734 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:08:41.789660 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-serving-cert: secret "maas-api-e2e-api-b-2eaba9-serving-cert" not found Aug 05 17:08:41.789734 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:08:41.789727 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls podName:f94f282a-021a-42b0-b5e0-7ee3db6c76f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:08:42.289711795 +0000 UTC m=+2753.749514054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls") pod "maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" (UID: "f94f282a-021a-42b0-b5e0-7ee3db6c76f1") : secret "maas-api-e2e-api-b-2eaba9-serving-cert" not found Aug 05 17:08:41.790347 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.790326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:41.797744 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:41.797721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhfb\" (UniqueName: \"kubernetes.io/projected/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-kube-api-access-mnhfb\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:42.293234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.293195 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6"] Aug 05 17:08:42.294328 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.294307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:42.296551 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.296533 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.296702 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.296682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls\") pod \"maas-api-e2e-api-b-2eaba9-7785945b7-tndmw\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:42.299074 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.299048 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2eaba9\"" Aug 05 17:08:42.308939 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.308912 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6"] Aug 05 17:08:42.317435 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.317414 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k"] Aug 05 17:08:42.321174 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.321153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.322946 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.322926 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2eaba9-dockercfg-s5wmq\"" Aug 05 17:08:42.329682 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.329659 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k"] Aug 05 17:08:42.394895 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.394854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt75\" (UniqueName: \"kubernetes.io/projected/6b35589c-a323-4794-9e97-aee391f0ed7a-kube-api-access-kzt75\") pod \"payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.395071 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.394899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180788bf-7f9a-4651-b74c-e8ae98356ce3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.395071 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.394939 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjk4\" (UniqueName: \"kubernetes.io/projected/180788bf-7f9a-4651-b74c-e8ae98356ce3-kube-api-access-7xjk4\") pod \"payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.395071 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.395028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b35589c-a323-4794-9e97-aee391f0ed7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.496410 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.496370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt75\" (UniqueName: \"kubernetes.io/projected/6b35589c-a323-4794-9e97-aee391f0ed7a-kube-api-access-kzt75\") pod \"payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.496410 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.496416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180788bf-7f9a-4651-b74c-e8ae98356ce3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.496620 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.496443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjk4\" (UniqueName: \"kubernetes.io/projected/180788bf-7f9a-4651-b74c-e8ae98356ce3-kube-api-access-7xjk4\") pod \"payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.496620 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.496477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b35589c-a323-4794-9e97-aee391f0ed7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.497150 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.497132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b35589c-a323-4794-9e97-aee391f0ed7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.497217 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.497170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180788bf-7f9a-4651-b74c-e8ae98356ce3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.503758 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.503728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjk4\" (UniqueName: \"kubernetes.io/projected/180788bf-7f9a-4651-b74c-e8ae98356ce3-kube-api-access-7xjk4\") pod \"payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.503904 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.503881 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt75\" (UniqueName: \"kubernetes.io/projected/6b35589c-a323-4794-9e97-aee391f0ed7a-kube-api-access-kzt75\") pod \"payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.534741 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.534705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:42.614127 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.614097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:42.632289 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.632264 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:42.670252 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.670025 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw"] Aug 05 17:08:42.673027 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:08:42.672556 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94f282a_021a_42b0_b5e0_7ee3db6c76f1.slice/crio-476fd4f5af10dcf57161c52197f954e4165063625e48cb4db58dcd94e1887b03 WatchSource:0}: Error finding container 476fd4f5af10dcf57161c52197f954e4165063625e48cb4db58dcd94e1887b03: Status 404 returned error can't find the container with id 476fd4f5af10dcf57161c52197f954e4165063625e48cb4db58dcd94e1887b03 Aug 05 17:08:42.761213 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.761175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" event={"ID":"f94f282a-021a-42b0-b5e0-7ee3db6c76f1","Type":"ContainerStarted","Data":"476fd4f5af10dcf57161c52197f954e4165063625e48cb4db58dcd94e1887b03"} Aug 05 17:08:42.798352 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.798324 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k"] Aug 05 17:08:42.800837 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:08:42.800809 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180788bf_7f9a_4651_b74c_e8ae98356ce3.slice/crio-6ab451fb931f2b6c5280caac34fce66491597a6bd565a58a604abb0e454bb76a WatchSource:0}: Error finding container 6ab451fb931f2b6c5280caac34fce66491597a6bd565a58a604abb0e454bb76a: Status 404 returned error can't find the container with id 6ab451fb931f2b6c5280caac34fce66491597a6bd565a58a604abb0e454bb76a Aug 05 17:08:42.801287 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:42.801267 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6"] Aug 05 17:08:42.801767 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:08:42.801746 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b35589c_a323_4794_9e97_aee391f0ed7a.slice/crio-f50c5612a6b827d540a2e9d5c3816fc518765b51f647027e41fc7d6300dc539d WatchSource:0}: Error finding container f50c5612a6b827d540a2e9d5c3816fc518765b51f647027e41fc7d6300dc539d: Status 404 returned error can't find the container with id f50c5612a6b827d540a2e9d5c3816fc518765b51f647027e41fc7d6300dc539d Aug 05 17:08:43.767715 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.767681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" event={"ID":"6b35589c-a323-4794-9e97-aee391f0ed7a","Type":"ContainerStarted","Data":"33744f0a4208a0f69ae3439d0bc721877257678728378f1c36b553d2eeb210c6"} Aug 05 17:08:43.768219 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.767738 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" event={"ID":"6b35589c-a323-4794-9e97-aee391f0ed7a","Type":"ContainerStarted","Data":"f50c5612a6b827d540a2e9d5c3816fc518765b51f647027e41fc7d6300dc539d"} Aug 05 17:08:43.768219 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.767788 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:08:43.769274 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.769255 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" event={"ID":"f94f282a-021a-42b0-b5e0-7ee3db6c76f1","Type":"ContainerStarted","Data":"b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689"} Aug 05 17:08:43.769400 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.769386 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:43.770747 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.770718 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" event={"ID":"180788bf-7f9a-4651-b74c-e8ae98356ce3","Type":"ContainerStarted","Data":"0afc273afb8f9c0f63ca81aade2452aa54fe94131191f894a09df24be2a4c279"} Aug 05 17:08:43.770871 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.770767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" event={"ID":"180788bf-7f9a-4651-b74c-e8ae98356ce3","Type":"ContainerStarted","Data":"6ab451fb931f2b6c5280caac34fce66491597a6bd565a58a604abb0e454bb76a"} Aug 05 17:08:43.770871 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.770811 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:08:43.785256 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.785219 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" podStartSLOduration=1.785205511 podStartE2EDuration="1.785205511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:08:43.783167973 +0000 UTC m=+2755.242970255" watchObservedRunningTime="2026-08-05 17:08:43.785205511 +0000 UTC m=+2755.245007790" Aug 05 17:08:43.800209 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.800172 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" podStartSLOduration=1.800161267 podStartE2EDuration="1.800161267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:08:43.798457855 +0000 UTC m=+2755.258260134" watchObservedRunningTime="2026-08-05 17:08:43.800161267 +0000 UTC m=+2755.259963546" Aug 05 17:08:43.816692 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:43.816648 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" podStartSLOduration=2.438253295 podStartE2EDuration="2.816634374s" totalImagesPullingTime="378.381079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:08:43.814275036 +0000 UTC m=+2755.274077315" watchObservedRunningTime="2026-08-05 17:08:43.816634374 +0000 UTC m=+2755.276436656" Aug 05 17:08:49.781598 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:49.781563 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:08:54.778384 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:54.778339 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" podUID="6b35589c-a323-4794-9e97-aee391f0ed7a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.58:9004: connect: connection refused" Aug 05 17:08:54.778868 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:08:54.778838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:09:04.777579 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:04.777547 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:09:24.384328 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.384145 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt"] Aug 05 17:09:24.384917 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.384700 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" podUID="c8db61b0-4243-4d22-95f3-845560dad361" containerName="maas-api" containerID="cri-o://63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b" gracePeriod=30 Aug 05 17:09:24.459640 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.459604 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg"] Aug 05 17:09:24.459908 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.459863 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" podUID="d3c0328a-b509-4032-b874-98950a178324" containerName="payload-processing" containerID="cri-o://25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d" gracePeriod=30 Aug 05 17:09:24.466062 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.466029 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s"] Aug 05 17:09:24.466285 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.466256 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" podUID="0848a915-d13e-461c-80ba-3e5777461a80" containerName="payload-pre-processing" containerID="cri-o://4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e" gracePeriod=30 Aug 05 17:09:24.620953 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.620907 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" podUID="c8db61b0-4243-4d22-95f3-845560dad361" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.54:8443/health\": dial tcp 10.133.0.54:8443: connect: connection refused" Aug 05 17:09:24.723546 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.723518 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:09:24.726700 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.726675 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:09:24.756985 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.756958 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:09:24.852466 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852426 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jkzd\" (UniqueName: \"kubernetes.io/projected/0848a915-d13e-461c-80ba-3e5777461a80-kube-api-access-8jkzd\") pod \"0848a915-d13e-461c-80ba-3e5777461a80\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " Aug 05 17:09:24.852656 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852478 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgnnj\" (UniqueName: \"kubernetes.io/projected/c8db61b0-4243-4d22-95f3-845560dad361-kube-api-access-cgnnj\") pod \"c8db61b0-4243-4d22-95f3-845560dad361\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " Aug 05 17:09:24.852656 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852533 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0848a915-d13e-461c-80ba-3e5777461a80-plugins-config-volume\") pod \"0848a915-d13e-461c-80ba-3e5777461a80\" (UID: \"0848a915-d13e-461c-80ba-3e5777461a80\") " Aug 05 17:09:24.852656 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852560 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db61b0-4243-4d22-95f3-845560dad361-openshift-service-ca\") pod \"c8db61b0-4243-4d22-95f3-845560dad361\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " Aug 05 17:09:24.852656 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852617 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db61b0-4243-4d22-95f3-845560dad361-maas-api-tls\") pod \"c8db61b0-4243-4d22-95f3-845560dad361\" (UID: \"c8db61b0-4243-4d22-95f3-845560dad361\") " Aug 05 17:09:24.852928 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852667 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxbdk\" (UniqueName: \"kubernetes.io/projected/d3c0328a-b509-4032-b874-98950a178324-kube-api-access-fxbdk\") pod \"d3c0328a-b509-4032-b874-98950a178324\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " Aug 05 17:09:24.852928 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852705 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c0328a-b509-4032-b874-98950a178324-plugins-config-volume\") pod \"d3c0328a-b509-4032-b874-98950a178324\" (UID: \"d3c0328a-b509-4032-b874-98950a178324\") " Aug 05 17:09:24.853033 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.852951 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0848a915-d13e-461c-80ba-3e5777461a80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0848a915-d13e-461c-80ba-3e5777461a80" (UID: "0848a915-d13e-461c-80ba-3e5777461a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:24.853033 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.853005 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8db61b0-4243-4d22-95f3-845560dad361-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8db61b0-4243-4d22-95f3-845560dad361" (UID: "c8db61b0-4243-4d22-95f3-845560dad361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:24.853279 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.853250 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c0328a-b509-4032-b874-98950a178324-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3c0328a-b509-4032-b874-98950a178324" (UID: "d3c0328a-b509-4032-b874-98950a178324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:24.854799 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.854770 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8db61b0-4243-4d22-95f3-845560dad361-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8db61b0-4243-4d22-95f3-845560dad361" (UID: "c8db61b0-4243-4d22-95f3-845560dad361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:09:24.854910 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.854829 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c0328a-b509-4032-b874-98950a178324-kube-api-access-fxbdk" (OuterVolumeSpecName: "kube-api-access-fxbdk") pod "d3c0328a-b509-4032-b874-98950a178324" (UID: "d3c0328a-b509-4032-b874-98950a178324"). InnerVolumeSpecName "kube-api-access-fxbdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:24.854910 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.854858 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8db61b0-4243-4d22-95f3-845560dad361-kube-api-access-cgnnj" (OuterVolumeSpecName: "kube-api-access-cgnnj") pod "c8db61b0-4243-4d22-95f3-845560dad361" (UID: "c8db61b0-4243-4d22-95f3-845560dad361"). InnerVolumeSpecName "kube-api-access-cgnnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:24.854992 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.854953 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0848a915-d13e-461c-80ba-3e5777461a80-kube-api-access-8jkzd" (OuterVolumeSpecName: "kube-api-access-8jkzd") pod "0848a915-d13e-461c-80ba-3e5777461a80" (UID: "0848a915-d13e-461c-80ba-3e5777461a80"). InnerVolumeSpecName "kube-api-access-8jkzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:24.954234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954145 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8db61b0-4243-4d22-95f3-845560dad361-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.954234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954176 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbdk\" (UniqueName: \"kubernetes.io/projected/d3c0328a-b509-4032-b874-98950a178324-kube-api-access-fxbdk\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.954234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954185 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c0328a-b509-4032-b874-98950a178324-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.954234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954195 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkzd\" (UniqueName: \"kubernetes.io/projected/0848a915-d13e-461c-80ba-3e5777461a80-kube-api-access-8jkzd\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.954234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954204 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgnnj\" (UniqueName: \"kubernetes.io/projected/c8db61b0-4243-4d22-95f3-845560dad361-kube-api-access-cgnnj\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.954234 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954234 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0848a915-d13e-461c-80ba-3e5777461a80-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.954547 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.954245 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8db61b0-4243-4d22-95f3-845560dad361-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:24.975062 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.975021 2567 generic.go:356] "Generic (PLEG): container finished" podID="0848a915-d13e-461c-80ba-3e5777461a80" containerID="4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e" exitCode=0 Aug 05 17:09:24.975259 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.975074 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" Aug 05 17:09:24.975259 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.975115 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" event={"ID":"0848a915-d13e-461c-80ba-3e5777461a80","Type":"ContainerDied","Data":"4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e"} Aug 05 17:09:24.975259 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.975170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s" event={"ID":"0848a915-d13e-461c-80ba-3e5777461a80","Type":"ContainerDied","Data":"ee4e0287b9c30831bffc9c08f12e030d0a76d4366db3d394431c76f626d9a8a4"} Aug 05 17:09:24.975259 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.975196 2567 scope.go:117] "RemoveContainer" containerID="4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e" Aug 05 17:09:24.976366 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.976342 2567 generic.go:356] "Generic (PLEG): container finished" podID="c8db61b0-4243-4d22-95f3-845560dad361" containerID="63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b" exitCode=0 Aug 05 17:09:24.976487 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.976405 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" Aug 05 17:09:24.976487 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.976429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" event={"ID":"c8db61b0-4243-4d22-95f3-845560dad361","Type":"ContainerDied","Data":"63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b"} Aug 05 17:09:24.976487 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.976474 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt" event={"ID":"c8db61b0-4243-4d22-95f3-845560dad361","Type":"ContainerDied","Data":"3d35b6775b127897b128dd44431df7c79746e6a4b22e561f6887910b32f2f105"} Aug 05 17:09:24.977836 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.977792 2567 generic.go:356] "Generic (PLEG): container finished" podID="d3c0328a-b509-4032-b874-98950a178324" containerID="25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d" exitCode=0 Aug 05 17:09:24.977947 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.977838 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" Aug 05 17:09:24.977947 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.977918 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" event={"ID":"d3c0328a-b509-4032-b874-98950a178324","Type":"ContainerDied","Data":"25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d"} Aug 05 17:09:24.978070 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.977977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg" event={"ID":"d3c0328a-b509-4032-b874-98950a178324","Type":"ContainerDied","Data":"da1c84cbe59a76a5e688fd78cb3a417bd58063656309f60c4d5685abc0be8eb0"} Aug 05 17:09:24.984656 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.984620 2567 scope.go:117] "RemoveContainer" containerID="4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e" Aug 05 17:09:24.984909 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:09:24.984885 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e\": container with ID starting with 4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e not found: ID does not exist" containerID="4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e" Aug 05 17:09:24.984998 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.984917 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e"} err="failed to get container status \"4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e\": rpc error: code = NotFound desc = could not find container \"4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e\": container with ID starting with 4abe75dd6d9b2a5d8a4ba60fa51bf00768670376be8efb927efe4e7deab7848e not found: ID does not exist" Aug 05 17:09:24.984998 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.984936 2567 scope.go:117] "RemoveContainer" containerID="63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b" Aug 05 17:09:24.993727 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.993708 2567 scope.go:117] "RemoveContainer" containerID="63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b" Aug 05 17:09:24.993996 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:09:24.993974 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b\": container with ID starting with 63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b not found: ID does not exist" containerID="63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b" Aug 05 17:09:24.994082 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.994002 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b"} err="failed to get container status \"63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b\": rpc error: code = NotFound desc = could not find container \"63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b\": container with ID starting with 63ba78113da152b46c102b7a4f27b2d9a720fd72da9c6a994a501766514c370b not found: ID does not exist" Aug 05 17:09:24.994082 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:24.994020 2567 scope.go:117] "RemoveContainer" containerID="25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d" Aug 05 17:09:25.002379 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.002290 2567 scope.go:117] "RemoveContainer" containerID="25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d" Aug 05 17:09:25.002667 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:09:25.002644 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d\": container with ID starting with 25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d not found: ID does not exist" containerID="25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d" Aug 05 17:09:25.002749 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.002675 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d"} err="failed to get container status \"25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d\": rpc error: code = NotFound desc = could not find container \"25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d\": container with ID starting with 25cb88f4a7824a3e728dc84f1309a5827ff37cd491ffc29cfdb7a752dbeabd9d not found: ID does not exist" Aug 05 17:09:25.004982 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.004964 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg"] Aug 05 17:09:25.012366 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.012342 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e153a-7b566999cd-q9nqg"] Aug 05 17:09:25.019643 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.019617 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s"] Aug 05 17:09:25.023995 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.023965 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e153a-746d6bfd8-mdg6s"] Aug 05 17:09:25.034242 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.034215 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt"] Aug 05 17:09:25.040978 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:25.040950 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e153a-587fd9bb9f-nlxdt"] Aug 05 17:09:27.010701 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:27.010668 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0848a915-d13e-461c-80ba-3e5777461a80" path="/var/lib/kubelet/pods/0848a915-d13e-461c-80ba-3e5777461a80/volumes" Aug 05 17:09:27.011115 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:27.011064 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8db61b0-4243-4d22-95f3-845560dad361" path="/var/lib/kubelet/pods/c8db61b0-4243-4d22-95f3-845560dad361/volumes" Aug 05 17:09:27.011405 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:27.011391 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c0328a-b509-4032-b874-98950a178324" path="/var/lib/kubelet/pods/d3c0328a-b509-4032-b874-98950a178324/volumes" Aug 05 17:09:42.752007 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:42.751965 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw"] Aug 05 17:09:42.752441 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:42.752319 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" podUID="f94f282a-021a-42b0-b5e0-7ee3db6c76f1" containerName="maas-api" containerID="cri-o://b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689" gracePeriod=30 Aug 05 17:09:42.861249 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:42.861213 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k"] Aug 05 17:09:42.861496 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:42.861437 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" podUID="180788bf-7f9a-4651-b74c-e8ae98356ce3" containerName="payload-processing" containerID="cri-o://0afc273afb8f9c0f63ca81aade2452aa54fe94131191f894a09df24be2a4c279" gracePeriod=30 Aug 05 17:09:42.875992 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:42.875960 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6"] Aug 05 17:09:42.876241 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:42.876191 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" podUID="6b35589c-a323-4794-9e97-aee391f0ed7a" containerName="payload-pre-processing" containerID="cri-o://33744f0a4208a0f69ae3439d0bc721877257678728378f1c36b553d2eeb210c6" gracePeriod=30 Aug 05 17:09:43.008880 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.008811 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:09:43.083121 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.083076 2567 generic.go:356] "Generic (PLEG): container finished" podID="6b35589c-a323-4794-9e97-aee391f0ed7a" containerID="33744f0a4208a0f69ae3439d0bc721877257678728378f1c36b553d2eeb210c6" exitCode=0 Aug 05 17:09:43.083246 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.083132 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" event={"ID":"6b35589c-a323-4794-9e97-aee391f0ed7a","Type":"ContainerDied","Data":"33744f0a4208a0f69ae3439d0bc721877257678728378f1c36b553d2eeb210c6"} Aug 05 17:09:43.084628 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.084606 2567 generic.go:356] "Generic (PLEG): container finished" podID="f94f282a-021a-42b0-b5e0-7ee3db6c76f1" containerID="b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689" exitCode=0 Aug 05 17:09:43.084733 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.084645 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" Aug 05 17:09:43.084733 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.084682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" event={"ID":"f94f282a-021a-42b0-b5e0-7ee3db6c76f1","Type":"ContainerDied","Data":"b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689"} Aug 05 17:09:43.084842 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.084739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw" event={"ID":"f94f282a-021a-42b0-b5e0-7ee3db6c76f1","Type":"ContainerDied","Data":"476fd4f5af10dcf57161c52197f954e4165063625e48cb4db58dcd94e1887b03"} Aug 05 17:09:43.084842 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.084770 2567 scope.go:117] "RemoveContainer" containerID="b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689" Aug 05 17:09:43.086750 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.086730 2567 generic.go:356] "Generic (PLEG): container finished" podID="180788bf-7f9a-4651-b74c-e8ae98356ce3" containerID="0afc273afb8f9c0f63ca81aade2452aa54fe94131191f894a09df24be2a4c279" exitCode=0 Aug 05 17:09:43.087411 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.086868 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" event={"ID":"180788bf-7f9a-4651-b74c-e8ae98356ce3","Type":"ContainerDied","Data":"0afc273afb8f9c0f63ca81aade2452aa54fe94131191f894a09df24be2a4c279"} Aug 05 17:09:43.095645 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.095622 2567 scope.go:117] "RemoveContainer" containerID="b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689" Aug 05 17:09:43.096007 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:09:43.095978 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689\": container with ID starting with b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689 not found: ID does not exist" containerID="b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689" Aug 05 17:09:43.096111 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.096021 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689"} err="failed to get container status \"b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689\": rpc error: code = NotFound desc = could not find container \"b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689\": container with ID starting with b47da0dcc00e33b128574c5d102d23fdb7b014fc7e7b3a1891be55a57ed5f689 not found: ID does not exist" Aug 05 17:09:43.149037 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.149014 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:09:43.151059 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.151039 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:09:43.209592 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209559 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b35589c-a323-4794-9e97-aee391f0ed7a-plugins-config-volume\") pod \"6b35589c-a323-4794-9e97-aee391f0ed7a\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " Aug 05 17:09:43.209592 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209600 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzt75\" (UniqueName: \"kubernetes.io/projected/6b35589c-a323-4794-9e97-aee391f0ed7a-kube-api-access-kzt75\") pod \"6b35589c-a323-4794-9e97-aee391f0ed7a\" (UID: \"6b35589c-a323-4794-9e97-aee391f0ed7a\") " Aug 05 17:09:43.209865 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209622 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180788bf-7f9a-4651-b74c-e8ae98356ce3-plugins-config-volume\") pod \"180788bf-7f9a-4651-b74c-e8ae98356ce3\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " Aug 05 17:09:43.209865 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209643 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls\") pod \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " Aug 05 17:09:43.209865 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209672 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnhfb\" (UniqueName: \"kubernetes.io/projected/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-kube-api-access-mnhfb\") pod \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " Aug 05 17:09:43.209865 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209691 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-openshift-service-ca\") pod \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\" (UID: \"f94f282a-021a-42b0-b5e0-7ee3db6c76f1\") " Aug 05 17:09:43.209865 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.209710 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xjk4\" (UniqueName: \"kubernetes.io/projected/180788bf-7f9a-4651-b74c-e8ae98356ce3-kube-api-access-7xjk4\") pod \"180788bf-7f9a-4651-b74c-e8ae98356ce3\" (UID: \"180788bf-7f9a-4651-b74c-e8ae98356ce3\") " Aug 05 17:09:43.210153 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.210042 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f94f282a-021a-42b0-b5e0-7ee3db6c76f1" (UID: "f94f282a-021a-42b0-b5e0-7ee3db6c76f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:43.210308 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.210272 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b35589c-a323-4794-9e97-aee391f0ed7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b35589c-a323-4794-9e97-aee391f0ed7a" (UID: "6b35589c-a323-4794-9e97-aee391f0ed7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:43.210795 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.210766 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180788bf-7f9a-4651-b74c-e8ae98356ce3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "180788bf-7f9a-4651-b74c-e8ae98356ce3" (UID: "180788bf-7f9a-4651-b74c-e8ae98356ce3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:43.212371 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.212346 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f94f282a-021a-42b0-b5e0-7ee3db6c76f1" (UID: "f94f282a-021a-42b0-b5e0-7ee3db6c76f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:09:43.212463 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.212355 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-kube-api-access-mnhfb" (OuterVolumeSpecName: "kube-api-access-mnhfb") pod "f94f282a-021a-42b0-b5e0-7ee3db6c76f1" (UID: "f94f282a-021a-42b0-b5e0-7ee3db6c76f1"). InnerVolumeSpecName "kube-api-access-mnhfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:43.212463 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.212373 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180788bf-7f9a-4651-b74c-e8ae98356ce3-kube-api-access-7xjk4" (OuterVolumeSpecName: "kube-api-access-7xjk4") pod "180788bf-7f9a-4651-b74c-e8ae98356ce3" (UID: "180788bf-7f9a-4651-b74c-e8ae98356ce3"). InnerVolumeSpecName "kube-api-access-7xjk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:43.212463 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.212381 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b35589c-a323-4794-9e97-aee391f0ed7a-kube-api-access-kzt75" (OuterVolumeSpecName: "kube-api-access-kzt75") pod "6b35589c-a323-4794-9e97-aee391f0ed7a" (UID: "6b35589c-a323-4794-9e97-aee391f0ed7a"). InnerVolumeSpecName "kube-api-access-kzt75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310293 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b35589c-a323-4794-9e97-aee391f0ed7a-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310340 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzt75\" (UniqueName: \"kubernetes.io/projected/6b35589c-a323-4794-9e97-aee391f0ed7a-kube-api-access-kzt75\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310353 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180788bf-7f9a-4651-b74c-e8ae98356ce3-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310365 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310379 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhfb\" (UniqueName: \"kubernetes.io/projected/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-kube-api-access-mnhfb\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310390 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94f282a-021a-42b0-b5e0-7ee3db6c76f1-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.310408 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.310402 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjk4\" (UniqueName: \"kubernetes.io/projected/180788bf-7f9a-4651-b74c-e8ae98356ce3-kube-api-access-7xjk4\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:09:43.405078 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.405044 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw"] Aug 05 17:09:43.409783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:43.409747 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eaba9-7785945b7-tndmw"] Aug 05 17:09:44.093214 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.093179 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" event={"ID":"6b35589c-a323-4794-9e97-aee391f0ed7a","Type":"ContainerDied","Data":"f50c5612a6b827d540a2e9d5c3816fc518765b51f647027e41fc7d6300dc539d"} Aug 05 17:09:44.093702 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.093226 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6" Aug 05 17:09:44.093702 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.093250 2567 scope.go:117] "RemoveContainer" containerID="33744f0a4208a0f69ae3439d0bc721877257678728378f1c36b553d2eeb210c6" Aug 05 17:09:44.095436 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.095404 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" event={"ID":"180788bf-7f9a-4651-b74c-e8ae98356ce3","Type":"ContainerDied","Data":"6ab451fb931f2b6c5280caac34fce66491597a6bd565a58a604abb0e454bb76a"} Aug 05 17:09:44.095436 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.095444 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k" Aug 05 17:09:44.102700 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.102677 2567 scope.go:117] "RemoveContainer" containerID="0afc273afb8f9c0f63ca81aade2452aa54fe94131191f894a09df24be2a4c279" Aug 05 17:09:44.118067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.118038 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6"] Aug 05 17:09:44.120508 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.120482 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2eaba9-9ff4c7c7b-d4vg6"] Aug 05 17:09:44.132894 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.132865 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k"] Aug 05 17:09:44.142608 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:44.142585 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2eaba9-65bb8566db-cg79k"] Aug 05 17:09:45.011763 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:45.011730 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180788bf-7f9a-4651-b74c-e8ae98356ce3" path="/var/lib/kubelet/pods/180788bf-7f9a-4651-b74c-e8ae98356ce3/volumes" Aug 05 17:09:45.012103 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:45.012073 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b35589c-a323-4794-9e97-aee391f0ed7a" path="/var/lib/kubelet/pods/6b35589c-a323-4794-9e97-aee391f0ed7a/volumes" Aug 05 17:09:45.012430 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:45.012418 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94f282a-021a-42b0-b5e0-7ee3db6c76f1" path="/var/lib/kubelet/pods/f94f282a-021a-42b0-b5e0-7ee3db6c76f1/volumes" Aug 05 17:09:55.336329 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336293 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27"] Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336731 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c0328a-b509-4032-b874-98950a178324" containerName="payload-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336748 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c0328a-b509-4032-b874-98950a178324" containerName="payload-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336757 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b35589c-a323-4794-9e97-aee391f0ed7a" containerName="payload-pre-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336763 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b35589c-a323-4794-9e97-aee391f0ed7a" containerName="payload-pre-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336773 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8db61b0-4243-4d22-95f3-845560dad361" containerName="maas-api" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336778 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8db61b0-4243-4d22-95f3-845560dad361" containerName="maas-api" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336791 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94f282a-021a-42b0-b5e0-7ee3db6c76f1" containerName="maas-api" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336796 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94f282a-021a-42b0-b5e0-7ee3db6c76f1" containerName="maas-api" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336803 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180788bf-7f9a-4651-b74c-e8ae98356ce3" containerName="payload-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336808 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="180788bf-7f9a-4651-b74c-e8ae98356ce3" containerName="payload-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336815 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0848a915-d13e-461c-80ba-3e5777461a80" containerName="payload-pre-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336820 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0848a915-d13e-461c-80ba-3e5777461a80" containerName="payload-pre-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336870 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="180788bf-7f9a-4651-b74c-e8ae98356ce3" containerName="payload-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336879 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94f282a-021a-42b0-b5e0-7ee3db6c76f1" containerName="maas-api" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336885 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0848a915-d13e-461c-80ba-3e5777461a80" containerName="payload-pre-processing" Aug 05 17:09:55.336886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336892 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c0328a-b509-4032-b874-98950a178324" containerName="payload-processing" Aug 05 17:09:55.337415 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336898 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b35589c-a323-4794-9e97-aee391f0ed7a" containerName="payload-pre-processing" Aug 05 17:09:55.337415 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.336904 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8db61b0-4243-4d22-95f3-845560dad361" containerName="maas-api" Aug 05 17:09:55.341072 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.341049 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.345054 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.345016 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-92434d-serving-cert\"" Aug 05 17:09:55.345211 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.345065 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 17:09:55.345466 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.345441 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 05 17:09:55.345621 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.345483 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8x4pl\"" Aug 05 17:09:55.359212 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.359121 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27"] Aug 05 17:09:55.401298 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.401251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c90177-fd60-458b-9206-0dec9a773a92-maas-api-tls\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.401497 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.401334 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhr5k\" (UniqueName: \"kubernetes.io/projected/a0c90177-fd60-458b-9206-0dec9a773a92-kube-api-access-xhr5k\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.401497 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.401365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c90177-fd60-458b-9206-0dec9a773a92-openshift-service-ca\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.502191 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.502156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhr5k\" (UniqueName: \"kubernetes.io/projected/a0c90177-fd60-458b-9206-0dec9a773a92-kube-api-access-xhr5k\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.502389 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.502199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c90177-fd60-458b-9206-0dec9a773a92-openshift-service-ca\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.502389 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.502238 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c90177-fd60-458b-9206-0dec9a773a92-maas-api-tls\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.502852 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.502821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c90177-fd60-458b-9206-0dec9a773a92-openshift-service-ca\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.504689 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.504664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c90177-fd60-458b-9206-0dec9a773a92-maas-api-tls\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.520636 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.520595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhr5k\" (UniqueName: \"kubernetes.io/projected/a0c90177-fd60-458b-9206-0dec9a773a92-kube-api-access-xhr5k\") pod \"maas-api-e2e-shared-a-92434d-68655649b7-gtd27\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.651351 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.651257 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:55.817027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:55.816989 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27"] Aug 05 17:09:55.819982 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:09:55.819945 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c90177_fd60_458b_9206_0dec9a773a92.slice/crio-d03792237529cba11214a92883c67cfaf793caddc25377a3952665e11463e1cc WatchSource:0}: Error finding container d03792237529cba11214a92883c67cfaf793caddc25377a3952665e11463e1cc: Status 404 returned error can't find the container with id d03792237529cba11214a92883c67cfaf793caddc25377a3952665e11463e1cc Aug 05 17:09:56.163443 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.163402 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" event={"ID":"a0c90177-fd60-458b-9206-0dec9a773a92","Type":"ContainerStarted","Data":"d03792237529cba11214a92883c67cfaf793caddc25377a3952665e11463e1cc"} Aug 05 17:09:56.343743 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.343712 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr"] Aug 05 17:09:56.348317 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.348293 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.350735 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.350706 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-92434d-dockercfg-pdx2b\"" Aug 05 17:09:56.350990 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.350974 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-92434d\"" Aug 05 17:09:56.366332 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.366298 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr"] Aug 05 17:09:56.410666 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.410634 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqbv\" (UniqueName: \"kubernetes.io/projected/1310d391-baf5-4364-9947-039460f939da-kube-api-access-kqqbv\") pod \"payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.410850 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.410684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1310d391-baf5-4364-9947-039460f939da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.429252 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.429170 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw"] Aug 05 17:09:56.432833 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.432807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.448073 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.448034 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw"] Aug 05 17:09:56.512058 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.512020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1310d391-baf5-4364-9947-039460f939da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.512266 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.512134 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8px4\" (UniqueName: \"kubernetes.io/projected/d306efbd-2020-4922-81ca-08bcf5a71851-kube-api-access-f8px4\") pod \"payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.512266 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.512177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqbv\" (UniqueName: \"kubernetes.io/projected/1310d391-baf5-4364-9947-039460f939da-kube-api-access-kqqbv\") pod \"payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.512266 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.512204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306efbd-2020-4922-81ca-08bcf5a71851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.512801 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.512780 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1310d391-baf5-4364-9947-039460f939da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.521046 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.521020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqbv\" (UniqueName: \"kubernetes.io/projected/1310d391-baf5-4364-9947-039460f939da-kube-api-access-kqqbv\") pod \"payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.613326 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.613279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8px4\" (UniqueName: \"kubernetes.io/projected/d306efbd-2020-4922-81ca-08bcf5a71851-kube-api-access-f8px4\") pod \"payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.613541 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.613344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306efbd-2020-4922-81ca-08bcf5a71851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.614037 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.614009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306efbd-2020-4922-81ca-08bcf5a71851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.621755 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.621734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8px4\" (UniqueName: \"kubernetes.io/projected/d306efbd-2020-4922-81ca-08bcf5a71851-kube-api-access-f8px4\") pod \"payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.666355 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.666317 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:56.744750 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.744716 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:56.814151 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.813146 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr"] Aug 05 17:09:56.816054 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:09:56.816023 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1310d391_baf5_4364_9947_039460f939da.slice/crio-1ec51593202aa222b9a47136bd70ec6628bd1df33bf91bf043cb4f1853794b52 WatchSource:0}: Error finding container 1ec51593202aa222b9a47136bd70ec6628bd1df33bf91bf043cb4f1853794b52: Status 404 returned error can't find the container with id 1ec51593202aa222b9a47136bd70ec6628bd1df33bf91bf043cb4f1853794b52 Aug 05 17:09:56.900745 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:56.900718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw"] Aug 05 17:09:56.902777 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:09:56.902752 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd306efbd_2020_4922_81ca_08bcf5a71851.slice/crio-8b4b2d4f3dcac7e06b9364079c09398b09860aeb067cac02790a6ba81f766bfb WatchSource:0}: Error finding container 8b4b2d4f3dcac7e06b9364079c09398b09860aeb067cac02790a6ba81f766bfb: Status 404 returned error can't find the container with id 8b4b2d4f3dcac7e06b9364079c09398b09860aeb067cac02790a6ba81f766bfb Aug 05 17:09:57.171029 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.170811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" event={"ID":"d306efbd-2020-4922-81ca-08bcf5a71851","Type":"ContainerStarted","Data":"23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414"} Aug 05 17:09:57.171029 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.170912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" event={"ID":"d306efbd-2020-4922-81ca-08bcf5a71851","Type":"ContainerStarted","Data":"8b4b2d4f3dcac7e06b9364079c09398b09860aeb067cac02790a6ba81f766bfb"} Aug 05 17:09:57.171029 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.170977 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:09:57.178295 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.178263 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" event={"ID":"a0c90177-fd60-458b-9206-0dec9a773a92","Type":"ContainerStarted","Data":"d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7"} Aug 05 17:09:57.178464 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.178328 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:09:57.179959 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.179937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" event={"ID":"1310d391-baf5-4364-9947-039460f939da","Type":"ContainerStarted","Data":"9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40"} Aug 05 17:09:57.180067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.179976 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" event={"ID":"1310d391-baf5-4364-9947-039460f939da","Type":"ContainerStarted","Data":"1ec51593202aa222b9a47136bd70ec6628bd1df33bf91bf043cb4f1853794b52"} Aug 05 17:09:57.180067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.179997 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:09:57.198079 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.198024 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" podStartSLOduration=1.198008433 podStartE2EDuration="1.198008433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:09:57.196002904 +0000 UTC m=+2828.655805186" watchObservedRunningTime="2026-08-05 17:09:57.198008433 +0000 UTC m=+2828.657810714" Aug 05 17:09:57.216858 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:09:57.216803 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" podStartSLOduration=1.7154357230000001 podStartE2EDuration="2.216784348s" totalImagesPullingTime="501.348625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:09:57.215083116 +0000 UTC m=+2828.674885397" watchObservedRunningTime="2026-08-05 17:09:57.216784348 +0000 UTC m=+2828.676586628" Aug 05 17:10:03.191534 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:03.191500 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:10:03.215196 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:03.215131 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" podStartSLOduration=7.215111307 podStartE2EDuration="7.215111307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:09:57.244681768 +0000 UTC m=+2828.704484050" watchObservedRunningTime="2026-08-05 17:10:03.215111307 +0000 UTC m=+2834.674913580" Aug 05 17:10:08.187746 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:08.187707 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:10:08.188169 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:08.187817 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:10:24.298830 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.298796 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww"] Aug 05 17:10:24.301970 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.301952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.303896 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.303871 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-57997e-serving-cert\"" Aug 05 17:10:24.317720 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.317692 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww"] Aug 05 17:10:24.343957 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.343924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tjr\" (UniqueName: \"kubernetes.io/projected/79954c67-5748-4b20-89e8-7ab6683343a9-kube-api-access-n6tjr\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.344120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.343963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79954c67-5748-4b20-89e8-7ab6683343a9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.344120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.344037 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.445143 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.445104 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.445295 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.445187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tjr\" (UniqueName: \"kubernetes.io/projected/79954c67-5748-4b20-89e8-7ab6683343a9-kube-api-access-n6tjr\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.445295 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.445209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79954c67-5748-4b20-89e8-7ab6683343a9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.445295 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:10:24.445247 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-serving-cert: secret "maas-api-e2e-shared-b-57997e-serving-cert" not found Aug 05 17:10:24.445405 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:10:24.445316 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls podName:79954c67-5748-4b20-89e8-7ab6683343a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:10:24.945299526 +0000 UTC m=+2856.405101788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls") pod "maas-api-e2e-shared-b-57997e-74c676499b-9xrww" (UID: "79954c67-5748-4b20-89e8-7ab6683343a9") : secret "maas-api-e2e-shared-b-57997e-serving-cert" not found Aug 05 17:10:24.445839 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.445822 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79954c67-5748-4b20-89e8-7ab6683343a9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.454222 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.454191 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tjr\" (UniqueName: \"kubernetes.io/projected/79954c67-5748-4b20-89e8-7ab6683343a9-kube-api-access-n6tjr\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.949517 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.949485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:24.951835 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:24.951809 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls\") pod \"maas-api-e2e-shared-b-57997e-74c676499b-9xrww\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:25.129781 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.129747 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9"] Aug 05 17:10:25.133373 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.133356 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.135499 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.135478 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-57997e-dockercfg-wmjph\"" Aug 05 17:10:25.135618 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.135523 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-57997e\"" Aug 05 17:10:25.142241 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.142218 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9"] Aug 05 17:10:25.216236 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.216152 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:25.224283 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.224255 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6"] Aug 05 17:10:25.228515 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.228495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.239105 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.239061 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6"] Aug 05 17:10:25.251907 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.251879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7pk\" (UniqueName: \"kubernetes.io/projected/f57a94f0-5049-406c-8f11-fa7a161a58ab-kube-api-access-np7pk\") pod \"payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.252041 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.251930 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a94f0-5049-406c-8f11-fa7a161a58ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.353149 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.353082 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7pk\" (UniqueName: \"kubernetes.io/projected/f57a94f0-5049-406c-8f11-fa7a161a58ab-kube-api-access-np7pk\") pod \"payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.353843 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.353166 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqldw\" (UniqueName: \"kubernetes.io/projected/1676ec35-41eb-43a8-ac4d-691326e0641d-kube-api-access-kqldw\") pod \"payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.353843 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.353283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a94f0-5049-406c-8f11-fa7a161a58ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.353843 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.353414 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676ec35-41eb-43a8-ac4d-691326e0641d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.354230 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.354210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a94f0-5049-406c-8f11-fa7a161a58ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.355365 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.355339 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww"] Aug 05 17:10:25.358246 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:10:25.358221 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79954c67_5748_4b20_89e8_7ab6683343a9.slice/crio-ca15cf26d47eaf42f22b68a6308c7a565869f582beb8fa0e7b038c0c8193faed WatchSource:0}: Error finding container ca15cf26d47eaf42f22b68a6308c7a565869f582beb8fa0e7b038c0c8193faed: Status 404 returned error can't find the container with id ca15cf26d47eaf42f22b68a6308c7a565869f582beb8fa0e7b038c0c8193faed Aug 05 17:10:25.361963 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.361943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7pk\" (UniqueName: \"kubernetes.io/projected/f57a94f0-5049-406c-8f11-fa7a161a58ab-kube-api-access-np7pk\") pod \"payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.445816 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.445779 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:25.454029 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.454004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqldw\" (UniqueName: \"kubernetes.io/projected/1676ec35-41eb-43a8-ac4d-691326e0641d-kube-api-access-kqldw\") pod \"payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.454166 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.454126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676ec35-41eb-43a8-ac4d-691326e0641d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.454769 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.454747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676ec35-41eb-43a8-ac4d-691326e0641d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.472106 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.472035 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqldw\" (UniqueName: \"kubernetes.io/projected/1676ec35-41eb-43a8-ac4d-691326e0641d-kube-api-access-kqldw\") pod \"payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.552561 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.552501 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:25.581818 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.581792 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9"] Aug 05 17:10:25.583134 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:10:25.583112 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57a94f0_5049_406c_8f11_fa7a161a58ab.slice/crio-38c81e325e33a81e82345145f1b11d0c35c998a560ec795d5228b81034a32050 WatchSource:0}: Error finding container 38c81e325e33a81e82345145f1b11d0c35c998a560ec795d5228b81034a32050: Status 404 returned error can't find the container with id 38c81e325e33a81e82345145f1b11d0c35c998a560ec795d5228b81034a32050 Aug 05 17:10:25.694481 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:25.694455 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6"] Aug 05 17:10:25.695631 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:10:25.695605 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1676ec35_41eb_43a8_ac4d_691326e0641d.slice/crio-e871cdfee5e6e256669d1a87ee58bf0c95c4a4955258328310b68b1b24cf99a9 WatchSource:0}: Error finding container e871cdfee5e6e256669d1a87ee58bf0c95c4a4955258328310b68b1b24cf99a9: Status 404 returned error can't find the container with id e871cdfee5e6e256669d1a87ee58bf0c95c4a4955258328310b68b1b24cf99a9 Aug 05 17:10:26.324562 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.324525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" event={"ID":"79954c67-5748-4b20-89e8-7ab6683343a9","Type":"ContainerStarted","Data":"1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349"} Aug 05 17:10:26.324779 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.324687 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:26.324779 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.324743 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" event={"ID":"79954c67-5748-4b20-89e8-7ab6683343a9","Type":"ContainerStarted","Data":"ca15cf26d47eaf42f22b68a6308c7a565869f582beb8fa0e7b038c0c8193faed"} Aug 05 17:10:26.326160 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.326133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" event={"ID":"f57a94f0-5049-406c-8f11-fa7a161a58ab","Type":"ContainerStarted","Data":"fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119"} Aug 05 17:10:26.326300 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.326177 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:10:26.326300 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.326203 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" event={"ID":"f57a94f0-5049-406c-8f11-fa7a161a58ab","Type":"ContainerStarted","Data":"38c81e325e33a81e82345145f1b11d0c35c998a560ec795d5228b81034a32050"} Aug 05 17:10:26.327795 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.327776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" event={"ID":"1676ec35-41eb-43a8-ac4d-691326e0641d","Type":"ContainerStarted","Data":"b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68"} Aug 05 17:10:26.327911 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.327807 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" event={"ID":"1676ec35-41eb-43a8-ac4d-691326e0641d","Type":"ContainerStarted","Data":"e871cdfee5e6e256669d1a87ee58bf0c95c4a4955258328310b68b1b24cf99a9"} Aug 05 17:10:26.327911 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.327840 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:26.342135 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.342076 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" podStartSLOduration=1.919328886 podStartE2EDuration="2.342062936s" totalImagesPullingTime="422.73405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:10:26.340395741 +0000 UTC m=+2857.800198024" watchObservedRunningTime="2026-08-05 17:10:26.342062936 +0000 UTC m=+2857.801865217" Aug 05 17:10:26.362627 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.362587 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" podStartSLOduration=1.362573721 podStartE2EDuration="1.362573721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:10:26.358886447 +0000 UTC m=+2857.818688728" watchObservedRunningTime="2026-08-05 17:10:26.362573721 +0000 UTC m=+2857.822376001" Aug 05 17:10:26.377794 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:26.377753 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" podStartSLOduration=1.377738849 podStartE2EDuration="1.377738849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:10:26.376592381 +0000 UTC m=+2857.836394659" watchObservedRunningTime="2026-08-05 17:10:26.377738849 +0000 UTC m=+2857.837541120" Aug 05 17:10:32.339200 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:32.339162 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:10:37.335746 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:37.335712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:10:37.336182 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:10:37.335842 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:12:49.170292 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:12:49.170262 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:12:49.179710 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:12:49.179688 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:17:49.210382 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:17:49.210357 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:17:49.217329 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:17:49.217311 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:19:42.435361 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.435279 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg"] Aug 05 17:19:42.438652 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.438632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.440484 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.440448 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0b8087-serving-cert\"" Aug 05 17:19:42.449451 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.449426 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg"] Aug 05 17:19:42.520664 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.520628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.520664 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.520663 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.520884 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.520693 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ncb\" (UniqueName: \"kubernetes.io/projected/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-kube-api-access-b2ncb\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.621625 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.621583 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.621625 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.621623 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.621920 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.621658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ncb\" (UniqueName: \"kubernetes.io/projected/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-kube-api-access-b2ncb\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.621920 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:19:42.621770 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-serving-cert: secret "maas-api-e2e-ipp-a-0b8087-serving-cert" not found Aug 05 17:19:42.621920 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:19:42.621845 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls podName:c7a49f7b-bada-468c-85d3-e3a92b51f6c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:43.121823884 +0000 UTC m=+3414.581626157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls") pod "maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" (UID: "c7a49f7b-bada-468c-85d3-e3a92b51f6c0") : secret "maas-api-e2e-ipp-a-0b8087-serving-cert" not found Aug 05 17:19:42.622311 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.622292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:42.630574 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:42.630549 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ncb\" (UniqueName: \"kubernetes.io/projected/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-kube-api-access-b2ncb\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:43.103612 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.103579 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h"] Aug 05 17:19:43.113147 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.113118 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.115283 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.115262 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0b8087\"" Aug 05 17:19:43.120837 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.120809 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h"] Aug 05 17:19:43.126429 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.126398 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:43.129190 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.129164 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:43.131286 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.131264 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg"] Aug 05 17:19:43.135161 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.135140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.139151 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.139127 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0b8087-dockercfg-jdshv\"" Aug 05 17:19:43.149972 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.149949 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg"] Aug 05 17:19:43.227053 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.227020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ec54e1-24fa-4831-8183-1165eeb64e50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.227240 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.227059 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c08332e-4933-4a90-9bab-9305c040ec98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.227240 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.227187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbqh\" (UniqueName: \"kubernetes.io/projected/8c08332e-4933-4a90-9bab-9305c040ec98-kube-api-access-twbqh\") pod \"payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.227240 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.227220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngl7\" (UniqueName: \"kubernetes.io/projected/17ec54e1-24fa-4831-8183-1165eeb64e50-kube-api-access-fngl7\") pod \"payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.328335 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.328299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngl7\" (UniqueName: \"kubernetes.io/projected/17ec54e1-24fa-4831-8183-1165eeb64e50-kube-api-access-fngl7\") pod \"payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.328510 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.328353 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ec54e1-24fa-4831-8183-1165eeb64e50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.328510 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.328385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c08332e-4933-4a90-9bab-9305c040ec98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.328510 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.328462 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbqh\" (UniqueName: \"kubernetes.io/projected/8c08332e-4933-4a90-9bab-9305c040ec98-kube-api-access-twbqh\") pod \"payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.329194 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.329175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ec54e1-24fa-4831-8183-1165eeb64e50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.329260 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.329240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c08332e-4933-4a90-9bab-9305c040ec98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.336358 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.336328 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngl7\" (UniqueName: \"kubernetes.io/projected/17ec54e1-24fa-4831-8183-1165eeb64e50-kube-api-access-fngl7\") pod \"payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.336529 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.336512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbqh\" (UniqueName: \"kubernetes.io/projected/8c08332e-4933-4a90-9bab-9305c040ec98-kube-api-access-twbqh\") pod \"payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.350312 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.350288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:43.425652 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.425621 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:43.446610 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.446581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:43.491911 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.491854 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg"] Aug 05 17:19:43.493075 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:19:43.493034 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a49f7b_bada_468c_85d3_e3a92b51f6c0.slice/crio-1e789acf7d5087b4c8eda3fe8939afd0fd57c3ac749b118a79fd571883e5a604 WatchSource:0}: Error finding container 1e789acf7d5087b4c8eda3fe8939afd0fd57c3ac749b118a79fd571883e5a604: Status 404 returned error can't find the container with id 1e789acf7d5087b4c8eda3fe8939afd0fd57c3ac749b118a79fd571883e5a604 Aug 05 17:19:43.495007 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.494985 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:19:43.579860 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.579793 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h"] Aug 05 17:19:43.581525 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:19:43.581461 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ec54e1_24fa_4831_8183_1165eeb64e50.slice/crio-ca42690bc1731e7d4eb62ff2a87797d135bd825fe8544865fa21f4f4129e7615 WatchSource:0}: Error finding container ca42690bc1731e7d4eb62ff2a87797d135bd825fe8544865fa21f4f4129e7615: Status 404 returned error can't find the container with id ca42690bc1731e7d4eb62ff2a87797d135bd825fe8544865fa21f4f4129e7615 Aug 05 17:19:43.812444 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:43.812415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg"] Aug 05 17:19:43.813682 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:19:43.813657 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c08332e_4933_4a90_9bab_9305c040ec98.slice/crio-ba3c43da7e1b03191935da93a6bc981b273eb3b4d56bc84887455893c54f2647 WatchSource:0}: Error finding container ba3c43da7e1b03191935da93a6bc981b273eb3b4d56bc84887455893c54f2647: Status 404 returned error can't find the container with id ba3c43da7e1b03191935da93a6bc981b273eb3b4d56bc84887455893c54f2647 Aug 05 17:19:44.089242 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.089210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" event={"ID":"17ec54e1-24fa-4831-8183-1165eeb64e50","Type":"ContainerStarted","Data":"8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9"} Aug 05 17:19:44.089427 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.089273 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:44.089427 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.089298 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" event={"ID":"17ec54e1-24fa-4831-8183-1165eeb64e50","Type":"ContainerStarted","Data":"ca42690bc1731e7d4eb62ff2a87797d135bd825fe8544865fa21f4f4129e7615"} Aug 05 17:19:44.090813 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.090783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" event={"ID":"c7a49f7b-bada-468c-85d3-e3a92b51f6c0","Type":"ContainerStarted","Data":"79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1"} Aug 05 17:19:44.090938 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.090837 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" event={"ID":"c7a49f7b-bada-468c-85d3-e3a92b51f6c0","Type":"ContainerStarted","Data":"1e789acf7d5087b4c8eda3fe8939afd0fd57c3ac749b118a79fd571883e5a604"} Aug 05 17:19:44.090938 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.090870 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:44.092330 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.092307 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" event={"ID":"8c08332e-4933-4a90-9bab-9305c040ec98","Type":"ContainerStarted","Data":"368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be"} Aug 05 17:19:44.092446 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.092367 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" event={"ID":"8c08332e-4933-4a90-9bab-9305c040ec98","Type":"ContainerStarted","Data":"ba3c43da7e1b03191935da93a6bc981b273eb3b4d56bc84887455893c54f2647"} Aug 05 17:19:44.092511 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.092441 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:19:44.106620 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.106573 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" podStartSLOduration=1.106558521 podStartE2EDuration="1.106558521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:44.105229571 +0000 UTC m=+3415.565031865" watchObservedRunningTime="2026-08-05 17:19:44.106558521 +0000 UTC m=+3415.566360855" Aug 05 17:19:44.122353 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.122264 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" podStartSLOduration=1.122249352 podStartE2EDuration="1.122249352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:44.120614518 +0000 UTC m=+3415.580416821" watchObservedRunningTime="2026-08-05 17:19:44.122249352 +0000 UTC m=+3415.582051661" Aug 05 17:19:44.136621 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:44.136468 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" podStartSLOduration=1.687780193 podStartE2EDuration="2.136449144s" totalImagesPullingTime="448.668951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:44.136407071 +0000 UTC m=+3415.596209365" watchObservedRunningTime="2026-08-05 17:19:44.136449144 +0000 UTC m=+3415.596251432" Aug 05 17:19:50.103680 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:50.103646 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:19:55.099850 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:55.099819 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:19:55.100313 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:19:55.099898 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:20:12.100689 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.100646 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r"] Aug 05 17:20:12.106104 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.106061 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.108763 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.108451 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ca1c80\"" Aug 05 17:20:12.111626 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.111600 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r"] Aug 05 17:20:12.141111 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.141065 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm"] Aug 05 17:20:12.144606 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.144590 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.146793 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.146773 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ca1c80-dockercfg-fwqb2\"" Aug 05 17:20:12.153889 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.153867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm"] Aug 05 17:20:12.189579 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.189549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2mb\" (UniqueName: \"kubernetes.io/projected/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-kube-api-access-mt2mb\") pod \"payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.189756 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.189596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a13cfc-eda1-4449-921d-a4ce45983a82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.189756 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.189654 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.189756 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.189707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/88a13cfc-eda1-4449-921d-a4ce45983a82-kube-api-access-p54qj\") pod \"payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.290735 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.290702 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/88a13cfc-eda1-4449-921d-a4ce45983a82-kube-api-access-p54qj\") pod \"payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.290921 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.290779 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2mb\" (UniqueName: \"kubernetes.io/projected/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-kube-api-access-mt2mb\") pod \"payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.290921 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.290815 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a13cfc-eda1-4449-921d-a4ce45983a82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.290921 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.290840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.291536 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.291511 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a13cfc-eda1-4449-921d-a4ce45983a82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.291613 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.291534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.299331 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.299302 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/88a13cfc-eda1-4449-921d-a4ce45983a82-kube-api-access-p54qj\") pod \"payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.299430 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.299413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2mb\" (UniqueName: \"kubernetes.io/projected/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-kube-api-access-mt2mb\") pod \"payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.417331 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.417233 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:12.457543 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.457464 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:12.565479 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.565322 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r"] Aug 05 17:20:12.568548 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:20:12.568519 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841b528e_05c0_4f46_8a64_b5fd7ecaa3b4.slice/crio-8c47567e7963f2c9ef9e0e8f4d4495bdcdb5a8f98ca0330eb64d5a356e2e9857 WatchSource:0}: Error finding container 8c47567e7963f2c9ef9e0e8f4d4495bdcdb5a8f98ca0330eb64d5a356e2e9857: Status 404 returned error can't find the container with id 8c47567e7963f2c9ef9e0e8f4d4495bdcdb5a8f98ca0330eb64d5a356e2e9857 Aug 05 17:20:12.813667 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:12.813635 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm"] Aug 05 17:20:12.815294 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:20:12.815265 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a13cfc_eda1_4449_921d_a4ce45983a82.slice/crio-030b4ab6a0cfe7ae8a7c3dc278b5dcfa6f84851831391055c84406b9fcf97e7c WatchSource:0}: Error finding container 030b4ab6a0cfe7ae8a7c3dc278b5dcfa6f84851831391055c84406b9fcf97e7c: Status 404 returned error can't find the container with id 030b4ab6a0cfe7ae8a7c3dc278b5dcfa6f84851831391055c84406b9fcf97e7c Aug 05 17:20:13.252724 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.252681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" event={"ID":"88a13cfc-eda1-4449-921d-a4ce45983a82","Type":"ContainerStarted","Data":"f5e32ba33f65c62b3c50b9881e18d1dafda7521fa576d7d1d0506e7c3f5579a5"} Aug 05 17:20:13.253221 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.252754 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:13.253221 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.252775 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" event={"ID":"88a13cfc-eda1-4449-921d-a4ce45983a82","Type":"ContainerStarted","Data":"030b4ab6a0cfe7ae8a7c3dc278b5dcfa6f84851831391055c84406b9fcf97e7c"} Aug 05 17:20:13.254226 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.254199 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" event={"ID":"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4","Type":"ContainerStarted","Data":"a74cb647d12a7668cb878cb308dfea1a8c3492edc2b4dd00ab263d3c2cf34102"} Aug 05 17:20:13.254318 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.254248 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" event={"ID":"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4","Type":"ContainerStarted","Data":"8c47567e7963f2c9ef9e0e8f4d4495bdcdb5a8f98ca0330eb64d5a356e2e9857"} Aug 05 17:20:13.254318 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.254304 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:13.269747 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.269695 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" podStartSLOduration=1.269679849 podStartE2EDuration="1.269679849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:13.267619353 +0000 UTC m=+3444.727421637" watchObservedRunningTime="2026-08-05 17:20:13.269679849 +0000 UTC m=+3444.729482129" Aug 05 17:20:13.282909 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:13.282854 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" podStartSLOduration=1.282839331 podStartE2EDuration="1.282839331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:13.28139723 +0000 UTC m=+3444.741199512" watchObservedRunningTime="2026-08-05 17:20:13.282839331 +0000 UTC m=+3444.742641612" Aug 05 17:20:24.261007 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:24.260965 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:20:24.261523 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:24.261307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:20:42.207350 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:20:42.207318 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm_88a13cfc-eda1-4449-921d-a4ce45983a82/payload-processing/0.log" Aug 05 17:21:32.060602 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:32.060525 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg_8c08332e-4933-4a90-9bab-9305c040ec98/payload-processing/0.log" Aug 05 17:21:32.300820 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:32.300789 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm_88a13cfc-eda1-4449-921d-a4ce45983a82/payload-processing/0.log" Aug 05 17:21:37.723275 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.723236 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc"] Aug 05 17:21:37.726755 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.726735 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:37.730070 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.730043 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-32fc6e-dockercfg-bvv8w\"" Aug 05 17:21:37.730246 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.730142 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-32fc6e\"" Aug 05 17:21:37.738320 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.738295 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc"] Aug 05 17:21:37.776972 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.776941 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7"] Aug 05 17:21:37.780276 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.780261 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:37.791507 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.791481 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7"] Aug 05 17:21:37.842634 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.842603 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtg2\" (UniqueName: \"kubernetes.io/projected/7fa3db46-afc3-4d6d-89c6-5042a5808f89-kube-api-access-cgtg2\") pod \"payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:37.842844 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.842640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3db46-afc3-4d6d-89c6-5042a5808f89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:37.943701 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.943670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdh9f\" (UniqueName: \"kubernetes.io/projected/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-kube-api-access-rdh9f\") pod \"payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:37.943701 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.943706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:37.943917 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.943769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtg2\" (UniqueName: \"kubernetes.io/projected/7fa3db46-afc3-4d6d-89c6-5042a5808f89-kube-api-access-cgtg2\") pod \"payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:37.943917 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.943805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3db46-afc3-4d6d-89c6-5042a5808f89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:37.944451 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.944433 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3db46-afc3-4d6d-89c6-5042a5808f89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:37.960812 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:37.960787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtg2\" (UniqueName: \"kubernetes.io/projected/7fa3db46-afc3-4d6d-89c6-5042a5808f89-kube-api-access-cgtg2\") pod \"payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:38.039464 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.039377 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:38.044726 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.044706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdh9f\" (UniqueName: \"kubernetes.io/projected/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-kube-api-access-rdh9f\") pod \"payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:38.044790 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.044737 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:38.045352 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.045332 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:38.053826 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.053801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdh9f\" (UniqueName: \"kubernetes.io/projected/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-kube-api-access-rdh9f\") pod \"payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:38.090896 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.090864 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:38.243965 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.243942 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7"] Aug 05 17:21:38.245992 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:21:38.245959 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3086c59_63f6_43b6_a1ae_e0c5dad42d49.slice/crio-4c5e7fbb67e6b5c5aaf2df2e056b757b93f8818b7d3e4922bb90a5dbe974571e WatchSource:0}: Error finding container 4c5e7fbb67e6b5c5aaf2df2e056b757b93f8818b7d3e4922bb90a5dbe974571e: Status 404 returned error can't find the container with id 4c5e7fbb67e6b5c5aaf2df2e056b757b93f8818b7d3e4922bb90a5dbe974571e Aug 05 17:21:38.387158 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.387132 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc"] Aug 05 17:21:38.388900 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:21:38.388872 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa3db46_afc3_4d6d_89c6_5042a5808f89.slice/crio-98321ed937e811b6d3043c3438686f36752ed6ae24d8c946ed907087b99c0663 WatchSource:0}: Error finding container 98321ed937e811b6d3043c3438686f36752ed6ae24d8c946ed907087b99c0663: Status 404 returned error can't find the container with id 98321ed937e811b6d3043c3438686f36752ed6ae24d8c946ed907087b99c0663 Aug 05 17:21:38.709701 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.709657 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" event={"ID":"d3086c59-63f6-43b6-a1ae-e0c5dad42d49","Type":"ContainerStarted","Data":"65d11ecf5174b7baf3ee8b34812e87f7a67a228b201526bc9803c19aa11cf2fd"} Aug 05 17:21:38.709909 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.709715 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" event={"ID":"d3086c59-63f6-43b6-a1ae-e0c5dad42d49","Type":"ContainerStarted","Data":"4c5e7fbb67e6b5c5aaf2df2e056b757b93f8818b7d3e4922bb90a5dbe974571e"} Aug 05 17:21:38.709909 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.709774 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:38.711119 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.711082 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" event={"ID":"7fa3db46-afc3-4d6d-89c6-5042a5808f89","Type":"ContainerStarted","Data":"098532c49b95ec588805e8af427e765fc4bc5158a1766bfeff3a364fd82256cf"} Aug 05 17:21:38.711227 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.711134 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" event={"ID":"7fa3db46-afc3-4d6d-89c6-5042a5808f89","Type":"ContainerStarted","Data":"98321ed937e811b6d3043c3438686f36752ed6ae24d8c946ed907087b99c0663"} Aug 05 17:21:38.711227 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.711153 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:21:38.726739 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.726686 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" podStartSLOduration=1.726672674 podStartE2EDuration="1.726672674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:38.723497813 +0000 UTC m=+3530.183300094" watchObservedRunningTime="2026-08-05 17:21:38.726672674 +0000 UTC m=+3530.186474954" Aug 05 17:21:38.740907 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:38.740855 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" podStartSLOduration=1.740835393 podStartE2EDuration="1.740835393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:38.740623359 +0000 UTC m=+3530.200425637" watchObservedRunningTime="2026-08-05 17:21:38.740835393 +0000 UTC m=+3530.200637673" Aug 05 17:21:49.717835 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:49.717805 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:21:49.718390 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:21:49.717890 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:22:07.646976 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.646937 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc"] Aug 05 17:22:07.647596 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.647180 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" podUID="7fa3db46-afc3-4d6d-89c6-5042a5808f89" containerName="payload-processing" containerID="cri-o://098532c49b95ec588805e8af427e765fc4bc5158a1766bfeff3a364fd82256cf" gracePeriod=30 Aug 05 17:22:07.655458 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.655287 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7"] Aug 05 17:22:07.655636 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.655508 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" podUID="d3086c59-63f6-43b6-a1ae-e0c5dad42d49" containerName="payload-pre-processing" containerID="cri-o://65d11ecf5174b7baf3ee8b34812e87f7a67a228b201526bc9803c19aa11cf2fd" gracePeriod=30 Aug 05 17:22:07.872789 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.872754 2567 generic.go:356] "Generic (PLEG): container finished" podID="d3086c59-63f6-43b6-a1ae-e0c5dad42d49" containerID="65d11ecf5174b7baf3ee8b34812e87f7a67a228b201526bc9803c19aa11cf2fd" exitCode=0 Aug 05 17:22:07.872900 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.872826 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" event={"ID":"d3086c59-63f6-43b6-a1ae-e0c5dad42d49","Type":"ContainerDied","Data":"65d11ecf5174b7baf3ee8b34812e87f7a67a228b201526bc9803c19aa11cf2fd"} Aug 05 17:22:07.874673 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.874645 2567 generic.go:356] "Generic (PLEG): container finished" podID="7fa3db46-afc3-4d6d-89c6-5042a5808f89" containerID="098532c49b95ec588805e8af427e765fc4bc5158a1766bfeff3a364fd82256cf" exitCode=0 Aug 05 17:22:07.874793 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.874718 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" event={"ID":"7fa3db46-afc3-4d6d-89c6-5042a5808f89","Type":"ContainerDied","Data":"098532c49b95ec588805e8af427e765fc4bc5158a1766bfeff3a364fd82256cf"} Aug 05 17:22:07.927692 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.927667 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:22:07.948034 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:07.948002 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:22:08.019434 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.019388 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3db46-afc3-4d6d-89c6-5042a5808f89-plugins-config-volume\") pod \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " Aug 05 17:22:08.019434 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.019444 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdh9f\" (UniqueName: \"kubernetes.io/projected/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-kube-api-access-rdh9f\") pod \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " Aug 05 17:22:08.019660 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.019497 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-plugins-config-volume\") pod \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\" (UID: \"d3086c59-63f6-43b6-a1ae-e0c5dad42d49\") " Aug 05 17:22:08.019660 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.019530 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgtg2\" (UniqueName: \"kubernetes.io/projected/7fa3db46-afc3-4d6d-89c6-5042a5808f89-kube-api-access-cgtg2\") pod \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\" (UID: \"7fa3db46-afc3-4d6d-89c6-5042a5808f89\") " Aug 05 17:22:08.019922 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.019888 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa3db46-afc3-4d6d-89c6-5042a5808f89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fa3db46-afc3-4d6d-89c6-5042a5808f89" (UID: "7fa3db46-afc3-4d6d-89c6-5042a5808f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:08.020052 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.019943 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3086c59-63f6-43b6-a1ae-e0c5dad42d49" (UID: "d3086c59-63f6-43b6-a1ae-e0c5dad42d49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:08.021643 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.021620 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-kube-api-access-rdh9f" (OuterVolumeSpecName: "kube-api-access-rdh9f") pod "d3086c59-63f6-43b6-a1ae-e0c5dad42d49" (UID: "d3086c59-63f6-43b6-a1ae-e0c5dad42d49"). InnerVolumeSpecName "kube-api-access-rdh9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:08.021714 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.021629 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa3db46-afc3-4d6d-89c6-5042a5808f89-kube-api-access-cgtg2" (OuterVolumeSpecName: "kube-api-access-cgtg2") pod "7fa3db46-afc3-4d6d-89c6-5042a5808f89" (UID: "7fa3db46-afc3-4d6d-89c6-5042a5808f89"). InnerVolumeSpecName "kube-api-access-cgtg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:08.120323 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.120287 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtg2\" (UniqueName: \"kubernetes.io/projected/7fa3db46-afc3-4d6d-89c6-5042a5808f89-kube-api-access-cgtg2\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:08.120323 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.120316 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa3db46-afc3-4d6d-89c6-5042a5808f89-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:08.120323 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.120325 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdh9f\" (UniqueName: \"kubernetes.io/projected/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-kube-api-access-rdh9f\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:08.120323 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.120333 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3086c59-63f6-43b6-a1ae-e0c5dad42d49-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:08.886490 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.886454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" event={"ID":"d3086c59-63f6-43b6-a1ae-e0c5dad42d49","Type":"ContainerDied","Data":"4c5e7fbb67e6b5c5aaf2df2e056b757b93f8818b7d3e4922bb90a5dbe974571e"} Aug 05 17:22:08.886940 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.886465 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7" Aug 05 17:22:08.886940 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.886555 2567 scope.go:117] "RemoveContainer" containerID="65d11ecf5174b7baf3ee8b34812e87f7a67a228b201526bc9803c19aa11cf2fd" Aug 05 17:22:08.888350 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.888324 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" Aug 05 17:22:08.888480 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.888343 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc" event={"ID":"7fa3db46-afc3-4d6d-89c6-5042a5808f89","Type":"ContainerDied","Data":"98321ed937e811b6d3043c3438686f36752ed6ae24d8c946ed907087b99c0663"} Aug 05 17:22:08.897681 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.897661 2567 scope.go:117] "RemoveContainer" containerID="098532c49b95ec588805e8af427e765fc4bc5158a1766bfeff3a364fd82256cf" Aug 05 17:22:08.908756 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.908728 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7"] Aug 05 17:22:08.915059 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.914999 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-32fc6e-6587b97d68-kfqw7"] Aug 05 17:22:08.924210 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.924180 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc"] Aug 05 17:22:08.928470 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:08.928443 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-32fc6e-587d4d59d6-z5zxc"] Aug 05 17:22:09.012056 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:09.012010 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa3db46-afc3-4d6d-89c6-5042a5808f89" path="/var/lib/kubelet/pods/7fa3db46-afc3-4d6d-89c6-5042a5808f89/volumes" Aug 05 17:22:09.012426 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:09.012412 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3086c59-63f6-43b6-a1ae-e0c5dad42d49" path="/var/lib/kubelet/pods/d3086c59-63f6-43b6-a1ae-e0c5dad42d49/volumes" Aug 05 17:22:27.192133 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.191947 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg"] Aug 05 17:22:27.262476 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.262444 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg"] Aug 05 17:22:27.262679 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.262651 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" podUID="8c08332e-4933-4a90-9bab-9305c040ec98" containerName="payload-processing" containerID="cri-o://368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be" gracePeriod=30 Aug 05 17:22:27.272331 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.272303 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h"] Aug 05 17:22:27.272590 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.272566 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" podUID="17ec54e1-24fa-4831-8183-1165eeb64e50" containerName="payload-pre-processing" containerID="cri-o://8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9" gracePeriod=30 Aug 05 17:22:27.563277 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.563251 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:22:27.566014 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.565992 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:22:27.698323 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.698284 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ec54e1-24fa-4831-8183-1165eeb64e50-plugins-config-volume\") pod \"17ec54e1-24fa-4831-8183-1165eeb64e50\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " Aug 05 17:22:27.698503 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.698375 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fngl7\" (UniqueName: \"kubernetes.io/projected/17ec54e1-24fa-4831-8183-1165eeb64e50-kube-api-access-fngl7\") pod \"17ec54e1-24fa-4831-8183-1165eeb64e50\" (UID: \"17ec54e1-24fa-4831-8183-1165eeb64e50\") " Aug 05 17:22:27.698503 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.698401 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twbqh\" (UniqueName: \"kubernetes.io/projected/8c08332e-4933-4a90-9bab-9305c040ec98-kube-api-access-twbqh\") pod \"8c08332e-4933-4a90-9bab-9305c040ec98\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " Aug 05 17:22:27.698503 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.698428 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c08332e-4933-4a90-9bab-9305c040ec98-plugins-config-volume\") pod \"8c08332e-4933-4a90-9bab-9305c040ec98\" (UID: \"8c08332e-4933-4a90-9bab-9305c040ec98\") " Aug 05 17:22:27.698879 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.698849 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ec54e1-24fa-4831-8183-1165eeb64e50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17ec54e1-24fa-4831-8183-1165eeb64e50" (UID: "17ec54e1-24fa-4831-8183-1165eeb64e50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:27.698959 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.698902 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c08332e-4933-4a90-9bab-9305c040ec98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c08332e-4933-4a90-9bab-9305c040ec98" (UID: "8c08332e-4933-4a90-9bab-9305c040ec98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:27.700641 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.700606 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c08332e-4933-4a90-9bab-9305c040ec98-kube-api-access-twbqh" (OuterVolumeSpecName: "kube-api-access-twbqh") pod "8c08332e-4933-4a90-9bab-9305c040ec98" (UID: "8c08332e-4933-4a90-9bab-9305c040ec98"). InnerVolumeSpecName "kube-api-access-twbqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:27.700757 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.700683 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ec54e1-24fa-4831-8183-1165eeb64e50-kube-api-access-fngl7" (OuterVolumeSpecName: "kube-api-access-fngl7") pod "17ec54e1-24fa-4831-8183-1165eeb64e50" (UID: "17ec54e1-24fa-4831-8183-1165eeb64e50"). InnerVolumeSpecName "kube-api-access-fngl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:27.799596 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.799503 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngl7\" (UniqueName: \"kubernetes.io/projected/17ec54e1-24fa-4831-8183-1165eeb64e50-kube-api-access-fngl7\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:27.799596 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.799536 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbqh\" (UniqueName: \"kubernetes.io/projected/8c08332e-4933-4a90-9bab-9305c040ec98-kube-api-access-twbqh\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:27.799596 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.799545 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c08332e-4933-4a90-9bab-9305c040ec98-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:27.799596 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.799554 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ec54e1-24fa-4831-8183-1165eeb64e50-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:27.987990 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.987959 2567 generic.go:356] "Generic (PLEG): container finished" podID="8c08332e-4933-4a90-9bab-9305c040ec98" containerID="368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be" exitCode=0 Aug 05 17:22:27.988178 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.988006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" Aug 05 17:22:27.988178 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.988066 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" event={"ID":"8c08332e-4933-4a90-9bab-9305c040ec98","Type":"ContainerDied","Data":"368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be"} Aug 05 17:22:27.988178 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.988151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg" event={"ID":"8c08332e-4933-4a90-9bab-9305c040ec98","Type":"ContainerDied","Data":"ba3c43da7e1b03191935da93a6bc981b273eb3b4d56bc84887455893c54f2647"} Aug 05 17:22:27.988339 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.988183 2567 scope.go:117] "RemoveContainer" containerID="368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be" Aug 05 17:22:27.989456 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.989429 2567 generic.go:356] "Generic (PLEG): container finished" podID="17ec54e1-24fa-4831-8183-1165eeb64e50" containerID="8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9" exitCode=0 Aug 05 17:22:27.989584 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.989478 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" Aug 05 17:22:27.989584 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.989528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" event={"ID":"17ec54e1-24fa-4831-8183-1165eeb64e50","Type":"ContainerDied","Data":"8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9"} Aug 05 17:22:27.989701 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.989579 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h" event={"ID":"17ec54e1-24fa-4831-8183-1165eeb64e50","Type":"ContainerDied","Data":"ca42690bc1731e7d4eb62ff2a87797d135bd825fe8544865fa21f4f4129e7615"} Aug 05 17:22:27.989760 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.989733 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" podUID="c7a49f7b-bada-468c-85d3-e3a92b51f6c0" containerName="maas-api" containerID="cri-o://79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1" gracePeriod=30 Aug 05 17:22:27.998766 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.998739 2567 scope.go:117] "RemoveContainer" containerID="368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be" Aug 05 17:22:27.999250 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:22:27.999228 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be\": container with ID starting with 368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be not found: ID does not exist" containerID="368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be" Aug 05 17:22:27.999339 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.999260 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be"} err="failed to get container status \"368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be\": rpc error: code = NotFound desc = could not find container \"368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be\": container with ID starting with 368728543d154f8f19d7b1e98a8560a3e794abed94497cf9f8d9c261571bf1be not found: ID does not exist" Aug 05 17:22:27.999339 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:27.999278 2567 scope.go:117] "RemoveContainer" containerID="8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9" Aug 05 17:22:28.007734 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.007714 2567 scope.go:117] "RemoveContainer" containerID="8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9" Aug 05 17:22:28.008017 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:22:28.007994 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9\": container with ID starting with 8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9 not found: ID does not exist" containerID="8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9" Aug 05 17:22:28.008100 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.008025 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9"} err="failed to get container status \"8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9\": rpc error: code = NotFound desc = could not find container \"8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9\": container with ID starting with 8512d2fbcdb1ad6ca69dbd4c010bda758332e5cb861288818944b557ab6609f9 not found: ID does not exist" Aug 05 17:22:28.028985 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.028951 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h"] Aug 05 17:22:28.035007 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.034981 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b8087-79f7d758d4-4dk6h"] Aug 05 17:22:28.044199 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.044172 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg"] Aug 05 17:22:28.050318 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.050258 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b8087-64cb6b5f5b-gzztg"] Aug 05 17:22:28.220941 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.220916 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:22:28.305063 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.304982 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-openshift-service-ca\") pod \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " Aug 05 17:22:28.305063 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.305036 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2ncb\" (UniqueName: \"kubernetes.io/projected/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-kube-api-access-b2ncb\") pod \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " Aug 05 17:22:28.305290 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.305070 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls\") pod \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\" (UID: \"c7a49f7b-bada-468c-85d3-e3a92b51f6c0\") " Aug 05 17:22:28.305436 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.305415 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7a49f7b-bada-468c-85d3-e3a92b51f6c0" (UID: "c7a49f7b-bada-468c-85d3-e3a92b51f6c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:28.307256 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.307233 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7a49f7b-bada-468c-85d3-e3a92b51f6c0" (UID: "c7a49f7b-bada-468c-85d3-e3a92b51f6c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:22:28.307320 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.307265 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-kube-api-access-b2ncb" (OuterVolumeSpecName: "kube-api-access-b2ncb") pod "c7a49f7b-bada-468c-85d3-e3a92b51f6c0" (UID: "c7a49f7b-bada-468c-85d3-e3a92b51f6c0"). InnerVolumeSpecName "kube-api-access-b2ncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:28.406767 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.406728 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2ncb\" (UniqueName: \"kubernetes.io/projected/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-kube-api-access-b2ncb\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:28.406767 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.406758 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:28.406990 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.406777 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a49f7b-bada-468c-85d3-e3a92b51f6c0-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:22:28.996465 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.996429 2567 generic.go:356] "Generic (PLEG): container finished" podID="c7a49f7b-bada-468c-85d3-e3a92b51f6c0" containerID="79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1" exitCode=0 Aug 05 17:22:28.996684 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.996499 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" Aug 05 17:22:28.996684 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.996528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" event={"ID":"c7a49f7b-bada-468c-85d3-e3a92b51f6c0","Type":"ContainerDied","Data":"79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1"} Aug 05 17:22:28.996684 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.996608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg" event={"ID":"c7a49f7b-bada-468c-85d3-e3a92b51f6c0","Type":"ContainerDied","Data":"1e789acf7d5087b4c8eda3fe8939afd0fd57c3ac749b118a79fd571883e5a604"} Aug 05 17:22:28.996684 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:28.996630 2567 scope.go:117] "RemoveContainer" containerID="79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1" Aug 05 17:22:29.006110 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:29.006073 2567 scope.go:117] "RemoveContainer" containerID="79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1" Aug 05 17:22:29.006459 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:22:29.006436 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1\": container with ID starting with 79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1 not found: ID does not exist" containerID="79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1" Aug 05 17:22:29.006519 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:29.006468 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1"} err="failed to get container status \"79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1\": rpc error: code = NotFound desc = could not find container \"79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1\": container with ID starting with 79a2319b9c220c1d502c962965e9b0cea2a1d2bff726d4cf492c5df064cd54d1 not found: ID does not exist" Aug 05 17:22:29.012008 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:29.011980 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ec54e1-24fa-4831-8183-1165eeb64e50" path="/var/lib/kubelet/pods/17ec54e1-24fa-4831-8183-1165eeb64e50/volumes" Aug 05 17:22:29.012350 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:29.012337 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c08332e-4933-4a90-9bab-9305c040ec98" path="/var/lib/kubelet/pods/8c08332e-4933-4a90-9bab-9305c040ec98/volumes" Aug 05 17:22:29.017357 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:29.017336 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg"] Aug 05 17:22:29.021840 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:29.021816 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b8087-6fcf6c76fd-ntnbg"] Aug 05 17:22:31.011772 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:31.011728 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a49f7b-bada-468c-85d3-e3a92b51f6c0" path="/var/lib/kubelet/pods/c7a49f7b-bada-468c-85d3-e3a92b51f6c0/volumes" Aug 05 17:22:49.242246 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:49.242218 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:22:49.249410 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:22:49.249391 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8hsb7_4c1229c1-ae68-4c83-8cde-7b1c7b2a4c64/ovn-acl-logging/0.log" Aug 05 17:23:27.095307 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.095273 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm"] Aug 05 17:23:27.095958 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.095523 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" podUID="88a13cfc-eda1-4449-921d-a4ce45983a82" containerName="payload-processing" containerID="cri-o://f5e32ba33f65c62b3c50b9881e18d1dafda7521fa576d7d1d0506e7c3f5579a5" gracePeriod=30 Aug 05 17:23:27.105316 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.105283 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r"] Aug 05 17:23:27.105618 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.105570 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" podUID="841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" containerName="payload-pre-processing" containerID="cri-o://a74cb647d12a7668cb878cb308dfea1a8c3492edc2b4dd00ab263d3c2cf34102" gracePeriod=30 Aug 05 17:23:27.298065 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.298028 2567 generic.go:356] "Generic (PLEG): container finished" podID="88a13cfc-eda1-4449-921d-a4ce45983a82" containerID="f5e32ba33f65c62b3c50b9881e18d1dafda7521fa576d7d1d0506e7c3f5579a5" exitCode=0 Aug 05 17:23:27.298244 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.298109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" event={"ID":"88a13cfc-eda1-4449-921d-a4ce45983a82","Type":"ContainerDied","Data":"f5e32ba33f65c62b3c50b9881e18d1dafda7521fa576d7d1d0506e7c3f5579a5"} Aug 05 17:23:27.299758 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.299737 2567 generic.go:356] "Generic (PLEG): container finished" podID="841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" containerID="a74cb647d12a7668cb878cb308dfea1a8c3492edc2b4dd00ab263d3c2cf34102" exitCode=0 Aug 05 17:23:27.299879 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.299766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" event={"ID":"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4","Type":"ContainerDied","Data":"a74cb647d12a7668cb878cb308dfea1a8c3492edc2b4dd00ab263d3c2cf34102"} Aug 05 17:23:27.390180 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.390156 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:23:27.401382 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.401362 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:23:27.509903 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.509871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-plugins-config-volume\") pod \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " Aug 05 17:23:27.509903 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.509904 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/88a13cfc-eda1-4449-921d-a4ce45983a82-kube-api-access-p54qj\") pod \"88a13cfc-eda1-4449-921d-a4ce45983a82\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " Aug 05 17:23:27.510145 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.509953 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt2mb\" (UniqueName: \"kubernetes.io/projected/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-kube-api-access-mt2mb\") pod \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\" (UID: \"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4\") " Aug 05 17:23:27.510145 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.510095 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a13cfc-eda1-4449-921d-a4ce45983a82-plugins-config-volume\") pod \"88a13cfc-eda1-4449-921d-a4ce45983a82\" (UID: \"88a13cfc-eda1-4449-921d-a4ce45983a82\") " Aug 05 17:23:27.510453 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.510421 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" (UID: "841b528e-05c0-4f46-8a64-b5fd7ecaa3b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:27.510565 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.510456 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a13cfc-eda1-4449-921d-a4ce45983a82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88a13cfc-eda1-4449-921d-a4ce45983a82" (UID: "88a13cfc-eda1-4449-921d-a4ce45983a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:27.512025 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.512003 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a13cfc-eda1-4449-921d-a4ce45983a82-kube-api-access-p54qj" (OuterVolumeSpecName: "kube-api-access-p54qj") pod "88a13cfc-eda1-4449-921d-a4ce45983a82" (UID: "88a13cfc-eda1-4449-921d-a4ce45983a82"). InnerVolumeSpecName "kube-api-access-p54qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:27.512110 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.512023 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-kube-api-access-mt2mb" (OuterVolumeSpecName: "kube-api-access-mt2mb") pod "841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" (UID: "841b528e-05c0-4f46-8a64-b5fd7ecaa3b4"). InnerVolumeSpecName "kube-api-access-mt2mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:27.611468 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.611397 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:27.611468 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.611422 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/88a13cfc-eda1-4449-921d-a4ce45983a82-kube-api-access-p54qj\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:27.611468 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.611438 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2mb\" (UniqueName: \"kubernetes.io/projected/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4-kube-api-access-mt2mb\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:27.611468 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:27.611448 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a13cfc-eda1-4449-921d-a4ce45983a82-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:28.307772 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.307744 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" Aug 05 17:23:28.308243 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.307741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm" event={"ID":"88a13cfc-eda1-4449-921d-a4ce45983a82","Type":"ContainerDied","Data":"030b4ab6a0cfe7ae8a7c3dc278b5dcfa6f84851831391055c84406b9fcf97e7c"} Aug 05 17:23:28.308243 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.307888 2567 scope.go:117] "RemoveContainer" containerID="f5e32ba33f65c62b3c50b9881e18d1dafda7521fa576d7d1d0506e7c3f5579a5" Aug 05 17:23:28.309261 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.309137 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" event={"ID":"841b528e-05c0-4f46-8a64-b5fd7ecaa3b4","Type":"ContainerDied","Data":"8c47567e7963f2c9ef9e0e8f4d4495bdcdb5a8f98ca0330eb64d5a356e2e9857"} Aug 05 17:23:28.309261 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.309182 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r" Aug 05 17:23:28.318256 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.318237 2567 scope.go:117] "RemoveContainer" containerID="a74cb647d12a7668cb878cb308dfea1a8c3492edc2b4dd00ab263d3c2cf34102" Aug 05 17:23:28.331076 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.331049 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm"] Aug 05 17:23:28.340207 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.340187 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca1c80-6d7d97646d-n8ggm"] Aug 05 17:23:28.350184 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.350158 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r"] Aug 05 17:23:28.356106 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:28.356074 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca1c80-6f944dc6d-8hp2r"] Aug 05 17:23:29.013506 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:29.013475 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" path="/var/lib/kubelet/pods/841b528e-05c0-4f46-8a64-b5fd7ecaa3b4/volumes" Aug 05 17:23:29.013886 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:29.013873 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a13cfc-eda1-4449-921d-a4ce45983a82" path="/var/lib/kubelet/pods/88a13cfc-eda1-4449-921d-a4ce45983a82/volumes" Aug 05 17:23:44.621244 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.621210 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27"] Aug 05 17:23:44.621803 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.621410 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" podUID="a0c90177-fd60-458b-9206-0dec9a773a92" containerName="maas-api" containerID="cri-o://d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7" gracePeriod=30 Aug 05 17:23:44.708002 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.707966 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw"] Aug 05 17:23:44.708231 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.708202 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" podUID="d306efbd-2020-4922-81ca-08bcf5a71851" containerName="payload-processing" containerID="cri-o://23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414" gracePeriod=30 Aug 05 17:23:44.715159 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.714513 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr"] Aug 05 17:23:44.715159 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.714902 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" podUID="1310d391-baf5-4364-9947-039460f939da" containerName="payload-pre-processing" containerID="cri-o://9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40" gracePeriod=30 Aug 05 17:23:44.973627 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.973596 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:23:44.981229 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.981206 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:23:44.993296 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:44.993273 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:23:45.063783 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063746 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhr5k\" (UniqueName: \"kubernetes.io/projected/a0c90177-fd60-458b-9206-0dec9a773a92-kube-api-access-xhr5k\") pod \"a0c90177-fd60-458b-9206-0dec9a773a92\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " Aug 05 17:23:45.063974 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063795 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqqbv\" (UniqueName: \"kubernetes.io/projected/1310d391-baf5-4364-9947-039460f939da-kube-api-access-kqqbv\") pod \"1310d391-baf5-4364-9947-039460f939da\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " Aug 05 17:23:45.063974 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063852 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c90177-fd60-458b-9206-0dec9a773a92-maas-api-tls\") pod \"a0c90177-fd60-458b-9206-0dec9a773a92\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " Aug 05 17:23:45.063974 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063877 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c90177-fd60-458b-9206-0dec9a773a92-openshift-service-ca\") pod \"a0c90177-fd60-458b-9206-0dec9a773a92\" (UID: \"a0c90177-fd60-458b-9206-0dec9a773a92\") " Aug 05 17:23:45.063974 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063897 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8px4\" (UniqueName: \"kubernetes.io/projected/d306efbd-2020-4922-81ca-08bcf5a71851-kube-api-access-f8px4\") pod \"d306efbd-2020-4922-81ca-08bcf5a71851\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " Aug 05 17:23:45.063974 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063930 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306efbd-2020-4922-81ca-08bcf5a71851-plugins-config-volume\") pod \"d306efbd-2020-4922-81ca-08bcf5a71851\" (UID: \"d306efbd-2020-4922-81ca-08bcf5a71851\") " Aug 05 17:23:45.063974 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.063957 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1310d391-baf5-4364-9947-039460f939da-plugins-config-volume\") pod \"1310d391-baf5-4364-9947-039460f939da\" (UID: \"1310d391-baf5-4364-9947-039460f939da\") " Aug 05 17:23:45.064335 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.064318 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c90177-fd60-458b-9206-0dec9a773a92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0c90177-fd60-458b-9206-0dec9a773a92" (UID: "a0c90177-fd60-458b-9206-0dec9a773a92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:45.064483 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.064454 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d306efbd-2020-4922-81ca-08bcf5a71851-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d306efbd-2020-4922-81ca-08bcf5a71851" (UID: "d306efbd-2020-4922-81ca-08bcf5a71851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:45.064667 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.064534 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1310d391-baf5-4364-9947-039460f939da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1310d391-baf5-4364-9947-039460f939da" (UID: "1310d391-baf5-4364-9947-039460f939da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:23:45.066201 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.066175 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c90177-fd60-458b-9206-0dec9a773a92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0c90177-fd60-458b-9206-0dec9a773a92" (UID: "a0c90177-fd60-458b-9206-0dec9a773a92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:23:45.066306 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.066232 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c90177-fd60-458b-9206-0dec9a773a92-kube-api-access-xhr5k" (OuterVolumeSpecName: "kube-api-access-xhr5k") pod "a0c90177-fd60-458b-9206-0dec9a773a92" (UID: "a0c90177-fd60-458b-9206-0dec9a773a92"). InnerVolumeSpecName "kube-api-access-xhr5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:45.066452 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.066432 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1310d391-baf5-4364-9947-039460f939da-kube-api-access-kqqbv" (OuterVolumeSpecName: "kube-api-access-kqqbv") pod "1310d391-baf5-4364-9947-039460f939da" (UID: "1310d391-baf5-4364-9947-039460f939da"). InnerVolumeSpecName "kube-api-access-kqqbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:45.066541 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.066519 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d306efbd-2020-4922-81ca-08bcf5a71851-kube-api-access-f8px4" (OuterVolumeSpecName: "kube-api-access-f8px4") pod "d306efbd-2020-4922-81ca-08bcf5a71851" (UID: "d306efbd-2020-4922-81ca-08bcf5a71851"). InnerVolumeSpecName "kube-api-access-f8px4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165328 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c90177-fd60-458b-9206-0dec9a773a92-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165357 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c90177-fd60-458b-9206-0dec9a773a92-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165367 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8px4\" (UniqueName: \"kubernetes.io/projected/d306efbd-2020-4922-81ca-08bcf5a71851-kube-api-access-f8px4\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165376 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306efbd-2020-4922-81ca-08bcf5a71851-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165385 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1310d391-baf5-4364-9947-039460f939da-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165393 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhr5k\" (UniqueName: \"kubernetes.io/projected/a0c90177-fd60-458b-9206-0dec9a773a92-kube-api-access-xhr5k\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.165412 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.165402 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqqbv\" (UniqueName: \"kubernetes.io/projected/1310d391-baf5-4364-9947-039460f939da-kube-api-access-kqqbv\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:23:45.398870 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.398838 2567 generic.go:356] "Generic (PLEG): container finished" podID="a0c90177-fd60-458b-9206-0dec9a773a92" containerID="d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7" exitCode=0 Aug 05 17:23:45.399076 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.398883 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" Aug 05 17:23:45.399076 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.398934 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" event={"ID":"a0c90177-fd60-458b-9206-0dec9a773a92","Type":"ContainerDied","Data":"d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7"} Aug 05 17:23:45.399076 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.398998 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27" event={"ID":"a0c90177-fd60-458b-9206-0dec9a773a92","Type":"ContainerDied","Data":"d03792237529cba11214a92883c67cfaf793caddc25377a3952665e11463e1cc"} Aug 05 17:23:45.399076 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.399028 2567 scope.go:117] "RemoveContainer" containerID="d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7" Aug 05 17:23:45.400271 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.400248 2567 generic.go:356] "Generic (PLEG): container finished" podID="1310d391-baf5-4364-9947-039460f939da" containerID="9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40" exitCode=0 Aug 05 17:23:45.400349 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.400290 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" Aug 05 17:23:45.400349 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.400327 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" event={"ID":"1310d391-baf5-4364-9947-039460f939da","Type":"ContainerDied","Data":"9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40"} Aug 05 17:23:45.400433 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.400371 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr" event={"ID":"1310d391-baf5-4364-9947-039460f939da","Type":"ContainerDied","Data":"1ec51593202aa222b9a47136bd70ec6628bd1df33bf91bf043cb4f1853794b52"} Aug 05 17:23:45.401662 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.401634 2567 generic.go:356] "Generic (PLEG): container finished" podID="d306efbd-2020-4922-81ca-08bcf5a71851" containerID="23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414" exitCode=0 Aug 05 17:23:45.401784 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.401669 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" Aug 05 17:23:45.401784 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.401692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" event={"ID":"d306efbd-2020-4922-81ca-08bcf5a71851","Type":"ContainerDied","Data":"23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414"} Aug 05 17:23:45.401784 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.401725 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw" event={"ID":"d306efbd-2020-4922-81ca-08bcf5a71851","Type":"ContainerDied","Data":"8b4b2d4f3dcac7e06b9364079c09398b09860aeb067cac02790a6ba81f766bfb"} Aug 05 17:23:45.410429 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.410404 2567 scope.go:117] "RemoveContainer" containerID="d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7" Aug 05 17:23:45.410708 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:23:45.410690 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7\": container with ID starting with d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7 not found: ID does not exist" containerID="d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7" Aug 05 17:23:45.410819 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.410714 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7"} err="failed to get container status \"d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7\": rpc error: code = NotFound desc = could not find container \"d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7\": container with ID starting with d29cbf347abb5df0e122128b41654424b002fc4cca3308c3e143f65e2cfafff7 not found: ID does not exist" Aug 05 17:23:45.410819 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.410729 2567 scope.go:117] "RemoveContainer" containerID="9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40" Aug 05 17:23:45.418810 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.418799 2567 scope.go:117] "RemoveContainer" containerID="9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40" Aug 05 17:23:45.419079 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:23:45.419064 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40\": container with ID starting with 9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40 not found: ID does not exist" containerID="9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40" Aug 05 17:23:45.419163 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.419099 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40"} err="failed to get container status \"9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40\": rpc error: code = NotFound desc = could not find container \"9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40\": container with ID starting with 9bebe6445e57c91b3aa4fe4438b0d3ebe0ff87567a5589fe39addc9259450e40 not found: ID does not exist" Aug 05 17:23:45.419163 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.419117 2567 scope.go:117] "RemoveContainer" containerID="23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414" Aug 05 17:23:45.422027 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.422006 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27"] Aug 05 17:23:45.425596 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.425573 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-92434d-68655649b7-gtd27"] Aug 05 17:23:45.427425 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.427406 2567 scope.go:117] "RemoveContainer" containerID="23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414" Aug 05 17:23:45.427666 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:23:45.427647 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414\": container with ID starting with 23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414 not found: ID does not exist" containerID="23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414" Aug 05 17:23:45.427711 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.427672 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414"} err="failed to get container status \"23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414\": rpc error: code = NotFound desc = could not find container \"23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414\": container with ID starting with 23a86d8f87fb822dab7c9bec235c2fc3b80cbb40f7e8623f766aeaa156487414 not found: ID does not exist" Aug 05 17:23:45.434986 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.434960 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw"] Aug 05 17:23:45.438511 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.438482 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-92434d-6f579b7b7d-s2wnw"] Aug 05 17:23:45.447587 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.447564 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr"] Aug 05 17:23:45.451457 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:45.451436 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-92434d-78c76847d4-lnptr"] Aug 05 17:23:47.010808 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:47.010767 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1310d391-baf5-4364-9947-039460f939da" path="/var/lib/kubelet/pods/1310d391-baf5-4364-9947-039460f939da/volumes" Aug 05 17:23:47.011208 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:47.011180 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c90177-fd60-458b-9206-0dec9a773a92" path="/var/lib/kubelet/pods/a0c90177-fd60-458b-9206-0dec9a773a92/volumes" Aug 05 17:23:47.011494 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:23:47.011480 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d306efbd-2020-4922-81ca-08bcf5a71851" path="/var/lib/kubelet/pods/d306efbd-2020-4922-81ca-08bcf5a71851/volumes" Aug 05 17:24:02.457041 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.457007 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww"] Aug 05 17:24:02.457544 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.457254 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" podUID="79954c67-5748-4b20-89e8-7ab6683343a9" containerName="maas-api" containerID="cri-o://1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349" gracePeriod=30 Aug 05 17:24:02.569960 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.569928 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6"] Aug 05 17:24:02.570209 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.570165 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" podUID="1676ec35-41eb-43a8-ac4d-691326e0641d" containerName="payload-processing" containerID="cri-o://b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68" gracePeriod=30 Aug 05 17:24:02.580300 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.580273 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9"] Aug 05 17:24:02.580543 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.580482 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" podUID="f57a94f0-5049-406c-8f11-fa7a161a58ab" containerName="payload-pre-processing" containerID="cri-o://fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119" gracePeriod=30 Aug 05 17:24:02.713191 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.713115 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:24:02.813584 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.813552 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls\") pod \"79954c67-5748-4b20-89e8-7ab6683343a9\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " Aug 05 17:24:02.813766 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.813592 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79954c67-5748-4b20-89e8-7ab6683343a9-openshift-service-ca\") pod \"79954c67-5748-4b20-89e8-7ab6683343a9\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " Aug 05 17:24:02.813766 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.813635 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6tjr\" (UniqueName: \"kubernetes.io/projected/79954c67-5748-4b20-89e8-7ab6683343a9-kube-api-access-n6tjr\") pod \"79954c67-5748-4b20-89e8-7ab6683343a9\" (UID: \"79954c67-5748-4b20-89e8-7ab6683343a9\") " Aug 05 17:24:02.814038 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.814011 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79954c67-5748-4b20-89e8-7ab6683343a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79954c67-5748-4b20-89e8-7ab6683343a9" (UID: "79954c67-5748-4b20-89e8-7ab6683343a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:24:02.817659 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.817628 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79954c67-5748-4b20-89e8-7ab6683343a9-kube-api-access-n6tjr" (OuterVolumeSpecName: "kube-api-access-n6tjr") pod "79954c67-5748-4b20-89e8-7ab6683343a9" (UID: "79954c67-5748-4b20-89e8-7ab6683343a9"). InnerVolumeSpecName "kube-api-access-n6tjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:24:02.817802 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.817783 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79954c67-5748-4b20-89e8-7ab6683343a9" (UID: "79954c67-5748-4b20-89e8-7ab6683343a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:24:02.835784 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.835762 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:24:02.842285 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.842263 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:24:02.914933 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.914901 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqldw\" (UniqueName: \"kubernetes.io/projected/1676ec35-41eb-43a8-ac4d-691326e0641d-kube-api-access-kqldw\") pod \"1676ec35-41eb-43a8-ac4d-691326e0641d\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " Aug 05 17:24:02.915116 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.914973 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np7pk\" (UniqueName: \"kubernetes.io/projected/f57a94f0-5049-406c-8f11-fa7a161a58ab-kube-api-access-np7pk\") pod \"f57a94f0-5049-406c-8f11-fa7a161a58ab\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " Aug 05 17:24:02.915116 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.914993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676ec35-41eb-43a8-ac4d-691326e0641d-plugins-config-volume\") pod \"1676ec35-41eb-43a8-ac4d-691326e0641d\" (UID: \"1676ec35-41eb-43a8-ac4d-691326e0641d\") " Aug 05 17:24:02.915116 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.915052 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a94f0-5049-406c-8f11-fa7a161a58ab-plugins-config-volume\") pod \"f57a94f0-5049-406c-8f11-fa7a161a58ab\" (UID: \"f57a94f0-5049-406c-8f11-fa7a161a58ab\") " Aug 05 17:24:02.915299 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.915196 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79954c67-5748-4b20-89e8-7ab6683343a9-maas-api-tls\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:02.915299 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.915207 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79954c67-5748-4b20-89e8-7ab6683343a9-openshift-service-ca\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:02.915299 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.915228 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tjr\" (UniqueName: \"kubernetes.io/projected/79954c67-5748-4b20-89e8-7ab6683343a9-kube-api-access-n6tjr\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:02.915455 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.915415 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1676ec35-41eb-43a8-ac4d-691326e0641d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1676ec35-41eb-43a8-ac4d-691326e0641d" (UID: "1676ec35-41eb-43a8-ac4d-691326e0641d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:24:02.915455 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.915424 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57a94f0-5049-406c-8f11-fa7a161a58ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f57a94f0-5049-406c-8f11-fa7a161a58ab" (UID: "f57a94f0-5049-406c-8f11-fa7a161a58ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:24:02.917010 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.916990 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57a94f0-5049-406c-8f11-fa7a161a58ab-kube-api-access-np7pk" (OuterVolumeSpecName: "kube-api-access-np7pk") pod "f57a94f0-5049-406c-8f11-fa7a161a58ab" (UID: "f57a94f0-5049-406c-8f11-fa7a161a58ab"). InnerVolumeSpecName "kube-api-access-np7pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:24:02.917139 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:02.917019 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1676ec35-41eb-43a8-ac4d-691326e0641d-kube-api-access-kqldw" (OuterVolumeSpecName: "kube-api-access-kqldw") pod "1676ec35-41eb-43a8-ac4d-691326e0641d" (UID: "1676ec35-41eb-43a8-ac4d-691326e0641d"). InnerVolumeSpecName "kube-api-access-kqldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:24:03.016120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.016017 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np7pk\" (UniqueName: \"kubernetes.io/projected/f57a94f0-5049-406c-8f11-fa7a161a58ab-kube-api-access-np7pk\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:03.016120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.016042 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676ec35-41eb-43a8-ac4d-691326e0641d-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:03.016120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.016053 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a94f0-5049-406c-8f11-fa7a161a58ab-plugins-config-volume\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:03.016120 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.016062 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqldw\" (UniqueName: \"kubernetes.io/projected/1676ec35-41eb-43a8-ac4d-691326e0641d-kube-api-access-kqldw\") on node \"ip-10-0-131-60.ec2.internal\" DevicePath \"\"" Aug 05 17:24:03.488069 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.488033 2567 generic.go:356] "Generic (PLEG): container finished" podID="1676ec35-41eb-43a8-ac4d-691326e0641d" containerID="b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68" exitCode=0 Aug 05 17:24:03.488069 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.488068 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" Aug 05 17:24:03.488562 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.488118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" event={"ID":"1676ec35-41eb-43a8-ac4d-691326e0641d","Type":"ContainerDied","Data":"b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68"} Aug 05 17:24:03.488562 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.488176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6" event={"ID":"1676ec35-41eb-43a8-ac4d-691326e0641d","Type":"ContainerDied","Data":"e871cdfee5e6e256669d1a87ee58bf0c95c4a4955258328310b68b1b24cf99a9"} Aug 05 17:24:03.488562 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.488209 2567 scope.go:117] "RemoveContainer" containerID="b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68" Aug 05 17:24:03.489569 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.489550 2567 generic.go:356] "Generic (PLEG): container finished" podID="79954c67-5748-4b20-89e8-7ab6683343a9" containerID="1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349" exitCode=0 Aug 05 17:24:03.489649 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.489616 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" Aug 05 17:24:03.489722 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.489613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" event={"ID":"79954c67-5748-4b20-89e8-7ab6683343a9","Type":"ContainerDied","Data":"1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349"} Aug 05 17:24:03.489785 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.489754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww" event={"ID":"79954c67-5748-4b20-89e8-7ab6683343a9","Type":"ContainerDied","Data":"ca15cf26d47eaf42f22b68a6308c7a565869f582beb8fa0e7b038c0c8193faed"} Aug 05 17:24:03.491046 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.491024 2567 generic.go:356] "Generic (PLEG): container finished" podID="f57a94f0-5049-406c-8f11-fa7a161a58ab" containerID="fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119" exitCode=0 Aug 05 17:24:03.491184 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.491067 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" Aug 05 17:24:03.491184 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.491120 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" event={"ID":"f57a94f0-5049-406c-8f11-fa7a161a58ab","Type":"ContainerDied","Data":"fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119"} Aug 05 17:24:03.491184 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.491168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9" event={"ID":"f57a94f0-5049-406c-8f11-fa7a161a58ab","Type":"ContainerDied","Data":"38c81e325e33a81e82345145f1b11d0c35c998a560ec795d5228b81034a32050"} Aug 05 17:24:03.499897 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.499879 2567 scope.go:117] "RemoveContainer" containerID="b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68" Aug 05 17:24:03.500166 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:24:03.500148 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68\": container with ID starting with b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68 not found: ID does not exist" containerID="b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68" Aug 05 17:24:03.500222 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.500175 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68"} err="failed to get container status \"b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68\": rpc error: code = NotFound desc = could not find container \"b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68\": container with ID starting with b37e70cd63c3df0b81b248676a6a895c45db776c6a83d625229c38b0bc757c68 not found: ID does not exist" Aug 05 17:24:03.500222 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.500191 2567 scope.go:117] "RemoveContainer" containerID="1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349" Aug 05 17:24:03.507984 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.507964 2567 scope.go:117] "RemoveContainer" containerID="1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349" Aug 05 17:24:03.508267 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:24:03.508247 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349\": container with ID starting with 1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349 not found: ID does not exist" containerID="1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349" Aug 05 17:24:03.508368 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.508270 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349"} err="failed to get container status \"1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349\": rpc error: code = NotFound desc = could not find container \"1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349\": container with ID starting with 1274741f64d6b8eb91ff440f655dd94cea9f16da2b25984e6eeaaf91c05f2349 not found: ID does not exist" Aug 05 17:24:03.508368 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.508288 2567 scope.go:117] "RemoveContainer" containerID="fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119" Aug 05 17:24:03.510583 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.510565 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9"] Aug 05 17:24:03.513551 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.513530 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-57997e-5b89c7d78d-wdkh9"] Aug 05 17:24:03.516586 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.516570 2567 scope.go:117] "RemoveContainer" containerID="fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119" Aug 05 17:24:03.516853 ip-10-0-131-60 kubenswrapper[2567]: E0805 17:24:03.516830 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119\": container with ID starting with fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119 not found: ID does not exist" containerID="fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119" Aug 05 17:24:03.516945 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.516858 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119"} err="failed to get container status \"fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119\": rpc error: code = NotFound desc = could not find container \"fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119\": container with ID starting with fe692632f7b61cc4f6c9a4307ded6ab674063ed95f6da9f469a565984be9e119 not found: ID does not exist" Aug 05 17:24:03.522820 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.522799 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6"] Aug 05 17:24:03.527741 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.527720 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-57997e-f6b54dc4b-vdjc6"] Aug 05 17:24:03.536197 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.536177 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww"] Aug 05 17:24:03.540640 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:03.540620 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57997e-74c676499b-9xrww"] Aug 05 17:24:05.012733 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:05.012689 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1676ec35-41eb-43a8-ac4d-691326e0641d" path="/var/lib/kubelet/pods/1676ec35-41eb-43a8-ac4d-691326e0641d/volumes" Aug 05 17:24:05.013362 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:05.013339 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79954c67-5748-4b20-89e8-7ab6683343a9" path="/var/lib/kubelet/pods/79954c67-5748-4b20-89e8-7ab6683343a9/volumes" Aug 05 17:24:05.013871 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:05.013849 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f57a94f0-5049-406c-8f11-fa7a161a58ab" path="/var/lib/kubelet/pods/f57a94f0-5049-406c-8f11-fa7a161a58ab/volumes" Aug 05 17:24:15.862059 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:15.861971 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4c85t_3e0b62e8-763c-40e0-b44b-20ba4aefdbad/server/0.log" Aug 05 17:24:15.988727 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:15.988697 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4zqsq_fd764260-49cf-46a1-885e-a3970fc15a17/manager/0.log" Aug 05 17:24:16.256272 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:16.256203 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-fmstf_0c0047bc-7177-4240-881e-a11afe1b29ba/manager/0.log" Aug 05 17:24:16.265936 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:16.265914 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-fmstf_0c0047bc-7177-4240-881e-a11afe1b29ba/copy-manifests/0.log" Aug 05 17:24:19.081789 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:19.081756 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xl9xk_864279c8-eb32-4d0b-b3c7-f1690e160f3f/discovery/0.log" Aug 05 17:24:20.108229 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:20.108192 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-27j5s_2cf1912d-e93d-479e-a0a9-c08af78d005e/main/0.log" Aug 05 17:24:20.337309 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:20.337283 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ck92j_16f37893-5aa0-4980-9da5-907fa48c2480/main/0.log" Aug 05 17:24:21.556757 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:21.556712 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-skwzj_936a1fd1-b818-429e-8551-9f3bd3afaf47/main/0.log" Aug 05 17:24:29.403595 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:29.403562 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kjwrw_5e464b10-9c26-458f-a3e6-5effcba181f5/global-pull-secret-syncer/0.log" Aug 05 17:24:29.501879 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:29.501842 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5ckqd_18a8caac-3a10-4067-acee-50b4d7bccf8e/konnectivity-agent/0.log" Aug 05 17:24:29.610204 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:29.610171 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-60.ec2.internal_39a7c3613bf8e5e7fd5e0d2f670dcc7b/haproxy/0.log" Aug 05 17:24:37.450118 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:37.450072 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpn8p_ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d/node-exporter/0.log" Aug 05 17:24:37.472230 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:37.472193 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpn8p_ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d/kube-rbac-proxy/0.log" Aug 05 17:24:37.495729 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:37.495705 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zpn8p_ead2f9ba-7a97-45dc-9dd1-11982a5e3a7d/init-textfile/0.log" Aug 05 17:24:39.513918 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.513878 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq"] Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514254 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c90177-fd60-458b-9206-0dec9a773a92" containerName="maas-api" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514270 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c90177-fd60-458b-9206-0dec9a773a92" containerName="maas-api" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514281 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ec54e1-24fa-4831-8183-1165eeb64e50" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514287 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ec54e1-24fa-4831-8183-1165eeb64e50" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514295 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1310d391-baf5-4364-9947-039460f939da" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514301 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1310d391-baf5-4364-9947-039460f939da" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514307 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d306efbd-2020-4922-81ca-08bcf5a71851" containerName="payload-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514313 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d306efbd-2020-4922-81ca-08bcf5a71851" containerName="payload-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514318 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3086c59-63f6-43b6-a1ae-e0c5dad42d49" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514326 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3086c59-63f6-43b6-a1ae-e0c5dad42d49" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514334 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa3db46-afc3-4d6d-89c6-5042a5808f89" containerName="payload-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514341 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa3db46-afc3-4d6d-89c6-5042a5808f89" containerName="payload-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514352 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514357 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" containerName="payload-pre-processing" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514366 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79954c67-5748-4b20-89e8-7ab6683343a9" containerName="maas-api" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514371 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="79954c67-5748-4b20-89e8-7ab6683343a9" containerName="maas-api" Aug 05 17:24:39.514369 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514377 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c08332e-4933-4a90-9bab-9305c040ec98" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514382 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c08332e-4933-4a90-9bab-9305c040ec98" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514392 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f57a94f0-5049-406c-8f11-fa7a161a58ab" containerName="payload-pre-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514398 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f57a94f0-5049-406c-8f11-fa7a161a58ab" containerName="payload-pre-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514403 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1676ec35-41eb-43a8-ac4d-691326e0641d" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514409 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1676ec35-41eb-43a8-ac4d-691326e0641d" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514418 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a49f7b-bada-468c-85d3-e3a92b51f6c0" containerName="maas-api" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514423 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a49f7b-bada-468c-85d3-e3a92b51f6c0" containerName="maas-api" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514430 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a13cfc-eda1-4449-921d-a4ce45983a82" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514435 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a13cfc-eda1-4449-921d-a4ce45983a82" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514487 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ec54e1-24fa-4831-8183-1165eeb64e50" containerName="payload-pre-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514495 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1676ec35-41eb-43a8-ac4d-691326e0641d" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514501 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f57a94f0-5049-406c-8f11-fa7a161a58ab" containerName="payload-pre-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514505 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1310d391-baf5-4364-9947-039460f939da" containerName="payload-pre-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514510 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c08332e-4933-4a90-9bab-9305c040ec98" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514515 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a49f7b-bada-468c-85d3-e3a92b51f6c0" containerName="maas-api" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514521 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a13cfc-eda1-4449-921d-a4ce45983a82" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514527 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c90177-fd60-458b-9206-0dec9a773a92" containerName="maas-api" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514531 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="79954c67-5748-4b20-89e8-7ab6683343a9" containerName="maas-api" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514536 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa3db46-afc3-4d6d-89c6-5042a5808f89" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514541 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d306efbd-2020-4922-81ca-08bcf5a71851" containerName="payload-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514547 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="841b528e-05c0-4f46-8a64-b5fd7ecaa3b4" containerName="payload-pre-processing" Aug 05 17:24:39.515067 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.514551 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3086c59-63f6-43b6-a1ae-e0c5dad42d49" containerName="payload-pre-processing" Aug 05 17:24:39.516767 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.516746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.518792 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.518765 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pbkcv\"/\"kube-root-ca.crt\"" Aug 05 17:24:39.519028 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.519009 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pbkcv\"/\"default-dockercfg-fv2mn\"" Aug 05 17:24:39.519148 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.519042 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pbkcv\"/\"openshift-service-ca.crt\"" Aug 05 17:24:39.530001 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.529977 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq"] Aug 05 17:24:39.617923 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.617879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-lib-modules\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.618135 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.617934 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-podres\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.618135 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.617954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-sys\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.618135 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.617978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsz7z\" (UniqueName: \"kubernetes.io/projected/82c906cf-a96d-4bf2-80fb-de74048c4f30-kube-api-access-fsz7z\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.618135 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.618054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-proc\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719102 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-podres\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719301 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-sys\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719301 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsz7z\" (UniqueName: \"kubernetes.io/projected/82c906cf-a96d-4bf2-80fb-de74048c4f30-kube-api-access-fsz7z\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719301 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-proc\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719301 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719218 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-podres\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719301 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-lib-modules\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719301 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719265 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-sys\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719520 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-proc\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.719520 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.719366 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82c906cf-a96d-4bf2-80fb-de74048c4f30-lib-modules\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.727632 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.727605 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsz7z\" (UniqueName: \"kubernetes.io/projected/82c906cf-a96d-4bf2-80fb-de74048c4f30-kube-api-access-fsz7z\") pod \"perf-node-gather-daemonset-92trq\" (UID: \"82c906cf-a96d-4bf2-80fb-de74048c4f30\") " pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.826756 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.826722 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:39.960713 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:39.960688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq"] Aug 05 17:24:39.962574 ip-10-0-131-60 kubenswrapper[2567]: W0805 17:24:39.962549 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82c906cf_a96d_4bf2_80fb_de74048c4f30.slice/crio-8569a862f23554da49c6ecbc03544ea67cffa7d9a2d282df1e01008382c707c3 WatchSource:0}: Error finding container 8569a862f23554da49c6ecbc03544ea67cffa7d9a2d282df1e01008382c707c3: Status 404 returned error can't find the container with id 8569a862f23554da49c6ecbc03544ea67cffa7d9a2d282df1e01008382c707c3 Aug 05 17:24:40.672131 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:40.672071 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" event={"ID":"82c906cf-a96d-4bf2-80fb-de74048c4f30","Type":"ContainerStarted","Data":"28c71d1c9810aaa17114d4d78f91647ea69baf416838ec22d11053ebf7625493"} Aug 05 17:24:40.672525 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:40.672157 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:40.672525 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:40.672180 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" event={"ID":"82c906cf-a96d-4bf2-80fb-de74048c4f30","Type":"ContainerStarted","Data":"8569a862f23554da49c6ecbc03544ea67cffa7d9a2d282df1e01008382c707c3"} Aug 05 17:24:40.689992 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:40.689935 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" podStartSLOduration=1.689919226 podStartE2EDuration="1.689919226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:40.689581739 +0000 UTC m=+3712.149384020" watchObservedRunningTime="2026-08-05 17:24:40.689919226 +0000 UTC m=+3712.149721507" Aug 05 17:24:42.479215 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:42.479188 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dht42_62cc78cf-1569-4119-9ad6-5fe13239a970/dns/0.log" Aug 05 17:24:42.500571 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:42.500543 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dht42_62cc78cf-1569-4119-9ad6-5fe13239a970/kube-rbac-proxy/0.log" Aug 05 17:24:42.628871 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:42.628838 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrpwk_6676a50f-361b-49f4-b68b-33549816ce4a/dns-node-resolver/0.log" Aug 05 17:24:43.279269 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:43.279235 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmlcr_cc776abf-676f-43e7-b4b5-7c68a29a7db7/node-ca/0.log" Aug 05 17:24:44.509631 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:44.509594 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xl9xk_864279c8-eb32-4d0b-b3c7-f1690e160f3f/discovery/0.log" Aug 05 17:24:45.654077 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:45.654047 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vv5rs_f0757141-204c-4f0d-b1af-5c515b125c42/serve-healthcheck-canary/0.log" Aug 05 17:24:46.169357 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:46.169329 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9pmrz_d5c4c891-bee5-45f8-8334-7607a36f95ff/kube-rbac-proxy/0.log" Aug 05 17:24:46.191395 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:46.191365 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9pmrz_d5c4c891-bee5-45f8-8334-7607a36f95ff/exporter/0.log" Aug 05 17:24:46.214379 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:46.214354 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9pmrz_d5c4c891-bee5-45f8-8334-7607a36f95ff/extractor/0.log" Aug 05 17:24:46.686399 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:46.686363 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pbkcv/perf-node-gather-daemonset-92trq" Aug 05 17:24:49.285349 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:49.285324 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4c85t_3e0b62e8-763c-40e0-b44b-20ba4aefdbad/server/0.log" Aug 05 17:24:49.364039 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:49.363985 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4zqsq_fd764260-49cf-46a1-885e-a3970fc15a17/manager/0.log" Aug 05 17:24:49.521855 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:49.521786 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-fmstf_0c0047bc-7177-4240-881e-a11afe1b29ba/manager/0.log" Aug 05 17:24:49.539931 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:49.539909 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfc779b6-fmstf_0c0047bc-7177-4240-881e-a11afe1b29ba/copy-manifests/0.log" Aug 05 17:24:50.490809 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:50.490778 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xl9xk_864279c8-eb32-4d0b-b3c7-f1690e160f3f/discovery/0.log" Aug 05 17:24:50.870015 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:50.869984 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bfcf8b475-kxtlw_168a0b8c-52d5-4f2c-9502-eed982eb37a3/manager/0.log" Aug 05 17:24:50.900973 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:50.900942 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-q2dtv_59b28568-c8ba-4bed-8bbc-4abb7e78c433/openshift-lws-operator/0.log" Aug 05 17:24:52.234188 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:52.234134 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xl9xk_864279c8-eb32-4d0b-b3c7-f1690e160f3f/discovery/0.log" Aug 05 17:24:55.926665 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:55.926637 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-pgbnl_86b0c9ea-bc97-4790-8a68-f8e19c07fa75/migrator/0.log" Aug 05 17:24:55.948657 ip-10-0-131-60 kubenswrapper[2567]: I0805 17:24:55.948629 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-pgbnl_86b0c9ea-bc97-4790-8a68-f8e19c07fa75/graceful-termination/0.log"