Aug 04 18:08:31.898496 ip-10-0-142-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 18:08:31.898510 ip-10-0-142-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 18:08:31.898519 ip-10-0-142-222 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 18:08:31.898854 ip-10-0-142-222 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 18:08:41.392346 ip-10-0-142-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Aug 04 18:08:41.392363 ip-10-0-142-222 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d4a1f57eea9c48eba24d6809806b28b4 -- Aug 04 18:10:34.737661 ip-10-0-142-222 systemd[1]: Starting Kubernetes Kubelet... Aug 04 18:10:35.153027 ip-10-0-142-222 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 18:10:35.153027 ip-10-0-142-222 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 18:10:35.153027 ip-10-0-142-222 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 18:10:35.153027 ip-10-0-142-222 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 18:10:35.153027 ip-10-0-142-222 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 18:10:35.153799 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.153125 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 18:10:35.157691 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157671 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:35.157691 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157690 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157694 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157698 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157702 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157705 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157712 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157715 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157718 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157721 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157724 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157727 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157729 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157732 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157734 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157737 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157740 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157742 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157745 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157747 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157750 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:35.157765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157752 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157755 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157757 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157760 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157763 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157766 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157769 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157771 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157775 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157778 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157782 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157785 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157788 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157791 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157793 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157796 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157798 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157801 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157804 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157807 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:35.158270 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157809 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157811 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157814 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157816 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157819 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157821 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157824 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157826 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157829 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157831 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157834 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157836 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157839 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157841 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157845 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157848 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157853 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157858 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157862 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:35.158765 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157866 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157868 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157871 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157874 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157877 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157880 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157883 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157885 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157888 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157890 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157893 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157895 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157898 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157901 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157904 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157908 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157911 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157914 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157917 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:35.159296 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157920 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157923 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157926 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157929 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157932 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157934 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157937 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157940 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157944 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157947 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157950 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157954 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157957 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157960 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157962 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157965 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157968 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157970 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157973 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.157976 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:35.160098 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158718 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158725 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158729 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158732 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158735 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158739 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158742 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158745 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158748 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158750 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158753 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158756 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158759 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158761 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158764 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158767 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158771 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158775 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158779 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158782 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:35.160740 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158786 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158789 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158792 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158794 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158797 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158800 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158803 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158805 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158808 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158811 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158813 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158816 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158818 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158821 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158823 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158826 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158828 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158831 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158834 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158836 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:35.161259 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158839 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158841 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158844 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158846 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158849 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158852 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158855 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158857 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158860 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158862 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158865 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158868 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158872 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158876 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158880 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158883 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158885 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158888 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158891 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:35.161776 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158894 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158896 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158899 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158902 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158904 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158907 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158909 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158912 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158916 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158918 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158921 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158924 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158927 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158929 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158932 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158935 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158937 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158940 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158942 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158944 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:35.162264 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158947 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158951 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158955 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158958 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158961 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158964 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158967 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158970 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158973 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158975 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158978 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158980 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158983 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158985 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158988 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158990 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158993 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158995 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.158998 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.159000 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:35.162869 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160393 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160416 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160423 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160427 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160430 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160433 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160437 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160444 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160447 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160450 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160454 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160460 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160463 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160466 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160469 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160472 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160476 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160480 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160485 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160488 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:35.163356 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160491 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160513 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160650 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160661 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160665 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160669 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160673 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160676 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160679 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160681 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160684 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160686 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160689 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160693 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160695 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160698 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160704 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160709 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160712 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:35.163895 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160715 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160719 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160722 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160725 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160728 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160730 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160733 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160735 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160738 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160741 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160744 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160746 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160749 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160752 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160755 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160759 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160761 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160764 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160767 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160769 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:35.164384 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160772 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160774 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160777 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160779 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160782 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160784 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160788 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160790 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160793 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160796 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160798 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160801 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160804 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160806 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160809 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160811 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160814 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160816 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160819 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:35.164863 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160822 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160824 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160827 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160829 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160832 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160834 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160837 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160840 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160843 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160845 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160848 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160851 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160853 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160856 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160858 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160861 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160864 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160867 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160869 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160874 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.160877 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:35.165385 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160954 2575 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160962 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160969 2575 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160974 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160978 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160982 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160986 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160991 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160994 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.160997 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161001 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161004 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161007 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161011 2575 flags.go:64] FLAG: --cgroup-root="" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161014 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161017 2575 flags.go:64] FLAG: --client-ca-file="" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161020 2575 flags.go:64] FLAG: --cloud-provider="external" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161024 2575 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161029 2575 flags.go:64] FLAG: --cluster-domain="" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161032 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161035 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161038 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161041 2575 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161045 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 18:10:35.165893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161048 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161051 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161055 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161058 2575 flags.go:64] FLAG: --contention-profiling="false" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161061 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161064 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161069 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161073 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161078 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161081 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161084 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161087 2575 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161090 2575 flags.go:64] FLAG: --enable-server="true" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161093 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161098 2575 flags.go:64] FLAG: --event-burst="100" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161101 2575 flags.go:64] FLAG: --event-qps="50" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161104 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161108 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161111 2575 flags.go:64] FLAG: --eviction-hard="" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161115 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161118 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161123 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161126 2575 flags.go:64] FLAG: --eviction-soft="" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161129 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161132 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 18:10:35.166491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161136 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161139 2575 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161142 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161145 2575 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161148 2575 flags.go:64] FLAG: --feature-gates="" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161152 2575 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161155 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161159 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161162 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161165 2575 flags.go:64] FLAG: --healthz-port="10248" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161169 2575 flags.go:64] FLAG: --help="false" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161172 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161175 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161179 2575 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161182 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161186 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161189 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161192 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161195 2575 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161198 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161201 2575 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161204 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161207 2575 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161210 2575 flags.go:64] FLAG: --kube-reserved="" Aug 04 18:10:35.167111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161213 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161216 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161219 2575 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161222 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161226 2575 flags.go:64] FLAG: --lock-file="" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161229 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161232 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161235 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161241 2575 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161245 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161248 2575 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161251 2575 flags.go:64] FLAG: --logging-format="text" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161253 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161257 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161260 2575 flags.go:64] FLAG: --manifest-url="" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161263 2575 flags.go:64] FLAG: --manifest-url-header="" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161267 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161271 2575 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161275 2575 flags.go:64] FLAG: --max-pods="110" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161278 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161281 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161286 2575 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161289 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161292 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161295 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 18:10:35.167744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161298 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161306 2575 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161309 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161313 2575 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161316 2575 flags.go:64] FLAG: --pod-cidr="" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161319 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161324 2575 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161327 2575 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161330 2575 flags.go:64] FLAG: --pods-per-core="0" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161333 2575 flags.go:64] FLAG: --port="10250" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161336 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161341 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aec38397f9ab4291" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161344 2575 flags.go:64] FLAG: --qos-reserved="" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161347 2575 flags.go:64] FLAG: --read-only-port="10255" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161350 2575 flags.go:64] FLAG: --register-node="true" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161353 2575 flags.go:64] FLAG: --register-with-taints="" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161357 2575 flags.go:64] FLAG: --registry-burst="10" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161377 2575 flags.go:64] FLAG: --registry-qps="5" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161381 2575 flags.go:64] FLAG: --reserved-cpus="" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161384 2575 flags.go:64] FLAG: --reserved-memory="" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161388 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161391 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161394 2575 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161398 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161401 2575 flags.go:64] FLAG: --runonce="false" Aug 04 18:10:35.168375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161403 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161407 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161410 2575 flags.go:64] FLAG: --seccomp-default="false" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161413 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161418 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161421 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161425 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161428 2575 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161431 2575 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161434 2575 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161437 2575 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161440 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161443 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161446 2575 flags.go:64] FLAG: --system-cgroups="" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161448 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161454 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161457 2575 flags.go:64] FLAG: --tls-cert-file="" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161460 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161466 2575 flags.go:64] FLAG: --tls-min-version="" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161468 2575 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161471 2575 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161474 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161477 2575 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161480 2575 flags.go:64] FLAG: --v="2" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161484 2575 flags.go:64] FLAG: --version="false" Aug 04 18:10:35.169017 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161489 2575 flags.go:64] FLAG: --vmodule="" Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161493 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161496 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161614 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161618 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161621 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161624 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161627 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161629 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161632 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161634 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161639 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161641 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161644 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161647 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161649 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161652 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161654 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161656 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:35.169713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161659 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161662 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161665 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161667 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161670 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161673 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161676 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161679 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161681 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161684 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161686 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161689 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161692 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161694 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161699 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161702 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161705 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161708 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161711 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161714 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:35.170179 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161716 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161719 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161721 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161724 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161727 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161730 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161733 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161736 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161738 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161742 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161746 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161749 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161752 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161755 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161758 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161760 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161763 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161765 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161769 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161772 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:35.170700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161775 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161777 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161781 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161783 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161786 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161789 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161792 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161795 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161798 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161800 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161803 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161805 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161808 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161811 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161813 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161816 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161821 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161828 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161831 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:35.171185 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161833 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161836 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161838 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161842 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161845 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161848 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161851 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161854 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161856 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161858 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161861 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161864 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161867 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161869 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161872 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161874 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161877 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161879 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161882 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161884 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:35.171658 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161887 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161890 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161893 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.161896 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.161902 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169056 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169072 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169091 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169097 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169154 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169159 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169163 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169166 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169169 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169171 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:35.172150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169174 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169176 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169179 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169182 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169184 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169187 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169189 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169192 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169195 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169197 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169200 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169202 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169205 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169207 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169210 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169212 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169215 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169217 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169222 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:35.172590 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169224 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169227 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169229 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169232 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169235 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169238 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169242 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169246 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169250 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169253 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169256 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169259 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169262 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169265 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169267 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169270 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169273 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169275 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169278 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:35.173075 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169280 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169283 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169285 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169288 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169290 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169293 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169295 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169298 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169301 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169303 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169306 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169309 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169312 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169315 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169318 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169321 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169323 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169326 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169329 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169333 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:35.173602 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169336 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169339 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169343 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169346 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169348 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169351 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169353 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169356 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169376 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169381 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169384 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169388 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169391 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169394 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169396 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169399 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169402 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169404 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169407 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169411 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:35.174142 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169415 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169418 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169421 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169424 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169426 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169429 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169433 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169435 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169438 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169441 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169444 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169447 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169449 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169452 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169454 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:35.174681 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169460 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169580 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169584 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169587 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169590 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169592 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169595 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169598 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169601 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169603 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169606 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169608 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169611 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169614 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169616 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169618 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169621 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169624 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169627 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 18:10:35.175057 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169629 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169632 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169635 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169638 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169640 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169648 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169651 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169653 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169656 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169659 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169661 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169664 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169666 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169670 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169673 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169676 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169679 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169681 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169684 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 18:10:35.175632 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169686 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169689 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169692 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169694 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169697 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169699 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169702 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169704 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169707 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169709 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169712 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169715 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169718 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169720 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169723 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169726 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169729 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169731 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169735 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169739 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 18:10:35.176110 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169742 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169745 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169749 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169752 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169754 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169757 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169760 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169763 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169766 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169768 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169770 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169773 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169775 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169778 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169780 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169783 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169786 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169788 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169791 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169793 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 18:10:35.176889 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169796 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169798 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169802 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169804 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169807 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169809 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169812 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169816 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169819 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169822 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169825 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169828 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169831 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169834 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169838 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169840 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169843 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169846 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169849 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169852 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 18:10:35.177641 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169854 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:35.169857 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169862 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.169983 2575 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.172049 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.173443 2575 server.go:1019] "Starting client certificate rotation" Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.173560 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.173605 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.175889 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 18:10:35.178267 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.177084 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 18:10:35.187349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.187325 2575 log.go:25] "Validated CRI v1 runtime API" Aug 04 18:10:35.192342 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.192326 2575 log.go:25] "Validated CRI v1 image API" Aug 04 18:10:35.194724 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.194697 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 18:10:35.197906 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.197883 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ba41c36-6ac5-42af-b66d-bcdffc92664f:/dev/nvme0n1p3 ab566c78-ad9c-4e13-ab55-b7b8a5aabaa8:/dev/nvme0n1p4] Aug 04 18:10:35.197985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.197906 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 18:10:35.203209 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.203083 2575 manager.go:216] Machine: {Timestamp:2026-08-04 18:10:35.20168314 +0000 UTC m=+0.348499391 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105297 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28715e7ca68f58b07e5486b8b88755 SystemUUID:ec28715e-7ca6-8f58-b07e-5486b8b88755 BootID:d4a1f57e-ea9c-48eb-a24d-6809806b28b4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:96:ac:64:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:96:ac:64:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:a4:18:b7:c4:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 18:10:35.203209 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.203199 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 18:10:35.203336 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.203301 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 18:10:35.206073 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.206053 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 18:10:35.206822 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.206785 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 18:10:35.206987 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.206826 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 18:10:35.208033 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.208018 2575 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 18:10:35.208033 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.208034 2575 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 18:10:35.208100 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.208043 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 18:10:35.208742 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.208728 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 18:10:35.210024 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.210008 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 18:10:35.214230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214211 2575 state_mem.go:36] "Initialized new in-memory state store" Aug 04 18:10:35.214390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214377 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 18:10:35.214430 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214418 2575 kubelet.go:486] "Attempting to sync node with API server" Aug 04 18:10:35.214465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214435 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 18:10:35.214465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214461 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 18:10:35.214519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214475 2575 kubelet.go:398] "Adding apiserver pod source" Aug 04 18:10:35.214519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214487 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 18:10:35.214880 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214868 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 18:10:35.214932 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.214886 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 18:10:35.221745 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.221720 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 18:10:35.223190 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.223175 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 18:10:35.223234 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.223207 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 18:10:35.224353 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224339 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 18:10:35.224415 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224380 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 18:10:35.224415 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224390 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 18:10:35.224415 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224399 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 18:10:35.224415 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224412 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224420 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224432 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224440 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224451 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224459 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224470 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 18:10:35.224519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.224478 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 18:10:35.225999 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.225985 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 18:10:35.226041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.226001 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 18:10:35.227758 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.227729 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 18:10:35.227796 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.227733 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 18:10:35.229553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.229532 2575 server.go:1268] "Started kubelet" Aug 04 18:10:35.229663 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.229634 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 18:10:35.229723 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.229661 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 18:10:35.229772 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.229731 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 18:10:35.230622 ip-10-0-142-222 systemd[1]: Started Kubernetes Kubelet. Aug 04 18:10:35.230919 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.230895 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 18:10:35.232532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.232514 2575 server.go:310] "Adding debug handlers to kubelet server" Aug 04 18:10:35.235962 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.235937 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 18:10:35.239163 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239148 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 18:10:35.239236 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239160 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 18:10:35.239290 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.239255 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.239714 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239691 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 18:10:35.239853 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239830 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 18:10:35.239969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239833 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 18:10:35.239969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239867 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 18:10:35.239969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239961 2575 reconstruct.go:97] "Volume reconstruction finished" Aug 04 18:10:35.240120 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.239973 2575 reconciler.go:29] "Reconciler: start to sync state" Aug 04 18:10:35.240120 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240040 2575 factory.go:55] Registering systemd factory Aug 04 18:10:35.240120 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240080 2575 factory.go:223] Registration of the systemd container factory successfully Aug 04 18:10:35.240346 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240326 2575 factory.go:153] Registering CRI-O factory Aug 04 18:10:35.240346 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240342 2575 factory.go:223] Registration of the crio container factory successfully Aug 04 18:10:35.240478 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240409 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 18:10:35.240478 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240433 2575 factory.go:103] Registering Raw factory Aug 04 18:10:35.240478 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240449 2575 manager.go:1196] Started watching for new ooms in manager Aug 04 18:10:35.240914 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.239744 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-222.ec2.internal.18c8ac9b4aef2af6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-222.ec2.internal,UID:ip-10-0-142-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-222.ec2.internal,},FirstTimestamp:2026-08-04 18:10:35.229498102 +0000 UTC m=+0.376314354,LastTimestamp:2026-08-04 18:10:35.229498102 +0000 UTC m=+0.376314354,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-222.ec2.internal,}" Aug 04 18:10:35.240914 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.240876 2575 manager.go:349] Starting recovery of all containers Aug 04 18:10:35.244927 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.244895 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 18:10:35.245120 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.245096 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 18:10:35.251474 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.251271 2575 manager.go:354] Recovery completed Aug 04 18:10:35.252924 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.252898 2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Aug 04 18:10:35.256214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.256197 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.262895 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.262873 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.262976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.262905 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.262976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.262917 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.263667 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.263648 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 18:10:35.263667 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.263664 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 18:10:35.263827 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.263681 2575 state_mem.go:36] "Initialized new in-memory state store" Aug 04 18:10:35.263827 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.263656 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.264725 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.264705 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.264806 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.264740 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.264806 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.264755 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.265658 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.265586 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-222.ec2.internal.18c8ac9b4cecb67c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-222.ec2.internal,UID:ip-10-0-142-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-222.ec2.internal,},FirstTimestamp:2026-08-04 18:10:35.262891644 +0000 UTC m=+0.409707896,LastTimestamp:2026-08-04 18:10:35.262891644 +0000 UTC m=+0.409707896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-222.ec2.internal,}" Aug 04 18:10:35.265942 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.265925 2575 policy_none.go:49] "None policy: Start" Aug 04 18:10:35.265942 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.265946 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 18:10:35.266078 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.265959 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 18:10:35.267927 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.267915 2575 policy_none.go:47] "Start" Aug 04 18:10:35.275957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.275936 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x58mg" Aug 04 18:10:35.276559 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.276491 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-222.ec2.internal.18c8ac9b4cecffbe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-222.ec2.internal,UID:ip-10-0-142-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-222.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-222.ec2.internal,},FirstTimestamp:2026-08-04 18:10:35.262910398 +0000 UTC m=+0.409726649,LastTimestamp:2026-08-04 18:10:35.262910398 +0000 UTC m=+0.409726649,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-222.ec2.internal,}" Aug 04 18:10:35.284956 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.284933 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x58mg" Aug 04 18:10:35.289089 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.289014 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-222.ec2.internal.18c8ac9b4ced288e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-222.ec2.internal,UID:ip-10-0-142-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-222.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-222.ec2.internal,},FirstTimestamp:2026-08-04 18:10:35.262920846 +0000 UTC m=+0.409737097,LastTimestamp:2026-08-04 18:10:35.262920846 +0000 UTC m=+0.409737097,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-222.ec2.internal,}" Aug 04 18:10:35.301953 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.301937 2575 manager.go:337] "Starting Device Plugin manager" Aug 04 18:10:35.302053 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.301981 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 18:10:35.302053 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.301991 2575 server.go:87] "Starting device plugin registration server" Aug 04 18:10:35.302289 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.302278 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 18:10:35.302323 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.302291 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 18:10:35.302406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.302393 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 18:10:35.302489 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.302478 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 18:10:35.302542 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.302491 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 18:10:35.303044 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.303021 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 18:10:35.303137 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.303076 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.340831 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.340802 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 18:10:35.347963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.342155 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 18:10:35.347963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.342190 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 18:10:35.347963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.342214 2575 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 18:10:35.347963 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.342246 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 18:10:35.347963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.344555 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:35.402576 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.402537 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.403589 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.403542 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.403589 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.403576 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.403589 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.403589 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.403729 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.403623 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.413159 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.413129 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.413159 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.413159 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-222.ec2.internal\": node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.442828 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.442789 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal"] Aug 04 18:10:35.443010 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.442881 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.443010 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.442934 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.444587 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.444571 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.444654 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.444601 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.444654 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.444613 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.446247 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.446235 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.446371 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.446346 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.446410 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.446396 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.447351 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.447331 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.447446 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.447378 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.447446 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.447394 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.447446 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.447408 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.447446 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.447427 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.447446 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.447436 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.449018 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.448997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.449018 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.449022 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 18:10:35.449838 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.449822 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientMemory" Aug 04 18:10:35.449924 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.449849 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 18:10:35.449924 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.449860 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeHasSufficientPID" Aug 04 18:10:35.469063 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.469041 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-222.ec2.internal\" not found" node="ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.473539 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.473524 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-222.ec2.internal\" not found" node="ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.541035 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.541009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa9c4aecedc139bae6541d09b9bc2bed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal\" (UID: \"aa9c4aecedc139bae6541d09b9bc2bed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.541149 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.541039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa9c4aecedc139bae6541d09b9bc2bed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal\" (UID: \"aa9c4aecedc139bae6541d09b9bc2bed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.541149 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.541059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90563c7e7eb521e876399cd564ee59a0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-222.ec2.internal\" (UID: \"90563c7e7eb521e876399cd564ee59a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.543042 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.543029 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.641850 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.641819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa9c4aecedc139bae6541d09b9bc2bed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal\" (UID: \"aa9c4aecedc139bae6541d09b9bc2bed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.641906 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.641851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa9c4aecedc139bae6541d09b9bc2bed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal\" (UID: \"aa9c4aecedc139bae6541d09b9bc2bed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.641906 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.641868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90563c7e7eb521e876399cd564ee59a0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-222.ec2.internal\" (UID: \"90563c7e7eb521e876399cd564ee59a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.641986 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.641924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90563c7e7eb521e876399cd564ee59a0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-222.ec2.internal\" (UID: \"90563c7e7eb521e876399cd564ee59a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.641986 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.641932 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa9c4aecedc139bae6541d09b9bc2bed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal\" (UID: \"aa9c4aecedc139bae6541d09b9bc2bed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.641986 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.641974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa9c4aecedc139bae6541d09b9bc2bed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal\" (UID: \"aa9c4aecedc139bae6541d09b9bc2bed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.643917 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.643901 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.744703 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.744633 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.770802 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.770779 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.776468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:35.776449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" Aug 04 18:10:35.845434 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.845397 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:35.945946 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:35.945926 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:36.046993 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.046914 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:36.147724 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.147699 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-222.ec2.internal\" not found" Aug 04 18:10:36.173845 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.173810 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 18:10:36.174479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.174018 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 18:10:36.194807 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.194784 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:36.215499 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.215477 2575 apiserver.go:52] "Watching apiserver" Aug 04 18:10:36.225725 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.225700 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 18:10:36.228078 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.228045 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2","openshift-cluster-node-tuning-operator/tuned-8x5kr","openshift-dns/node-resolver-tjjg9","openshift-image-registry/node-ca-xn6wp","openshift-network-diagnostics/network-check-target-fwqwx","openshift-network-operator/iptables-alerter-2w24j","openshift-multus/multus-additional-cni-plugins-c896x","openshift-multus/multus-w7w54","openshift-multus/network-metrics-daemon-zv8cr","openshift-ovn-kubernetes/ovnkube-node-pcjf4","kube-system/global-pull-secret-syncer-4fkfc","kube-system/konnectivity-agent-h6dbd"] Aug 04 18:10:36.231249 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.231233 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.233978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.233958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.234068 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.234021 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:36.234307 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.234290 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-52wr9\"" Aug 04 18:10:36.234392 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.234296 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 18:10:36.234478 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.234338 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.234536 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.234481 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 18:10:36.234605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.234589 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.237264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.237231 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.239316 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.239289 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" Aug 04 18:10:36.239420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.239320 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 18:10:36.239616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.239587 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zf8p8\"" Aug 04 18:10:36.239764 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.239747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.239871 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.239849 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 18:10:36.240144 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.240126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 18:10:36.242527 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.242507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.243803 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.243788 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 18:10:36.244259 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.244246 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.245520 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.245606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysctl-conf\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.245606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8f68d2-8865-437d-b247-afaac34a9474-cni-binary-copy\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.245606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-socket-dir-parent\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.245757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-cni-multus\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.245757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-host\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.245757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/195010ea-cabb-4497-9b46-c46ab80c6137-etc-tuned\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.245757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-hostroot\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.245757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-kubernetes\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.245757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-systemd\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-netns\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-cni-bin\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-kubelet\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prj6g\" (UniqueName: \"kubernetes.io/projected/9a17151a-4a1c-4988-95f7-8a3b293cf489-kube-api-access-prj6g\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-socket-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-device-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysconfig\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-system-cni-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.245999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-os-release\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246015 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-conf-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc8f68d2-8865-437d-b247-afaac34a9474-multus-daemon-config\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.246091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51c8b78-a028-4ace-8861-369f61795c3b-konnectivity-ca\") pod \"konnectivity-agent-h6dbd\" (UID: \"b51c8b78-a028-4ace-8861-369f61795c3b\") " pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-sys-fs\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-cni-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v2qq4\"" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-cnibin\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246238 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-k8s-cni-cncf-io\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246283 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-run\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-sys\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-lib-modules\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppzl\" (UniqueName: \"kubernetes.io/projected/195010ea-cabb-4497-9b46-c46ab80c6137-kube-api-access-lppzl\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-etc-kubernetes\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7vf\" (UniqueName: \"kubernetes.io/projected/fc8f68d2-8865-437d-b247-afaac34a9474-kube-api-access-bp7vf\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-registration-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-modprobe-d\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysctl-d\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.246856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246573 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-twwwp\"" Aug 04 18:10:36.247469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-var-lib-kubelet\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.247469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51c8b78-a028-4ace-8861-369f61795c3b-agent-certs\") pod \"konnectivity-agent-h6dbd\" (UID: \"b51c8b78-a028-4ace-8861-369f61795c3b\") " pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.247469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4gz\" (UniqueName: \"kubernetes.io/projected/44dd09e6-1d0f-4ea6-8637-4531305f5c46-kube-api-access-nm4gz\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.247469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195010ea-cabb-4497-9b46-c46ab80c6137-tmp\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.247469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.246687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-multus-certs\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.248856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.248839 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.249488 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.249470 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.249845 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.249826 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j2n4w\"" Aug 04 18:10:36.249950 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.249825 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.251155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.251139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:36.251224 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.251194 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:36.252908 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.252889 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.252981 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.252955 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 18:10:36.253064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.253044 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.253389 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.253371 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w2w9k\"" Aug 04 18:10:36.253594 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.253570 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.256201 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.256184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.256661 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.256643 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.256757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.256715 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.256819 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.256762 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 18:10:36.257040 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.257013 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p6v29\"" Aug 04 18:10:36.257170 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.257155 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 18:10:36.257252 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.257240 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" Aug 04 18:10:36.258744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.258710 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 18:10:36.258850 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.258775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.259180 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.259160 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 18:10:36.259324 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.259291 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2ztn\"" Aug 04 18:10:36.260783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.260768 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 18:10:36.261242 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.261221 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal"] Aug 04 18:10:36.261349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.261305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.261411 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.261352 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:36.262294 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262277 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 18:10:36.262337 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 18:10:36.262370 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262359 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 18:10:36.262538 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262522 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 18:10:36.262600 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262543 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-988kc\"" Aug 04 18:10:36.262655 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262623 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 18:10:36.262711 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.262694 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 18:10:36.280905 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.278941 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal"] Aug 04 18:10:36.281303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.281283 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 18:10:36.289880 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.289828 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 18:05:35 +0000 UTC" deadline="2028-03-22 04:56:30.313162843 +0000 UTC" Aug 04 18:10:36.289880 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.289876 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14290h45m54.023288671s" Aug 04 18:10:36.302359 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.302306 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tdmm6" Aug 04 18:10:36.311907 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.311885 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tdmm6" Aug 04 18:10:36.340957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.340933 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 18:10:36.346914 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.346889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-kubelet\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347031 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.346923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-device-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.347031 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.346958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.347031 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.346973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-kubelet\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347031 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.346984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-sys\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.347031 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-lib-modules\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-sys\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-device-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-system-cni-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-conf-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-lib-modules\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc8f68d2-8865-437d-b247-afaac34a9474-multus-daemon-config\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51c8b78-a028-4ace-8861-369f61795c3b-konnectivity-ca\") pod \"konnectivity-agent-h6dbd\" (UID: \"b51c8b78-a028-4ace-8861-369f61795c3b\") " pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-conf-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-cni-bin\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-k8s-cni-cncf-io\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-system-cni-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-run\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-run\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-ovnkube-config\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-k8s-cni-cncf-io\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.347293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-registration-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-registration-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-modprobe-d\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqk9\" (UniqueName: \"kubernetes.io/projected/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-kube-api-access-4jqk9\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cnibin\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-modprobe-d\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-etc-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/098d0be6-e403-4144-ab99-3835833be835-ovn-node-metrics-cert\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51c8b78-a028-4ace-8861-369f61795c3b-agent-certs\") pod \"konnectivity-agent-h6dbd\" (UID: \"b51c8b78-a028-4ace-8861-369f61795c3b\") " pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4gz\" (UniqueName: \"kubernetes.io/projected/44dd09e6-1d0f-4ea6-8637-4531305f5c46-kube-api-access-nm4gz\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195010ea-cabb-4497-9b46-c46ab80c6137-tmp\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-tmp-dir\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-multus-certs\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysctl-conf\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.348150 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-host-slash\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8f68d2-8865-437d-b247-afaac34a9474-cni-binary-copy\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc8f68d2-8865-437d-b247-afaac34a9474-multus-daemon-config\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-cni-multus\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-multus-certs\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-cni-multus\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347876 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347955 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysctl-conf\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.347989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/195010ea-cabb-4497-9b46-c46ab80c6137-etc-tuned\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-iptables-alerter-script\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-kubelet\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-ovn\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-log-socket\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-hostroot\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2d36207-38b9-4ace-a64d-f0439dea7bdc-dbus\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-host\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-hostroot\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw624\" (UniqueName: \"kubernetes.io/projected/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-kube-api-access-gw624\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.348951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-os-release\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6jd\" (UniqueName: \"kubernetes.io/projected/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-kube-api-access-hk6jd\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc8f68d2-8865-437d-b247-afaac34a9474-cni-binary-copy\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-cni-bin\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prj6g\" (UniqueName: \"kubernetes.io/projected/9a17151a-4a1c-4988-95f7-8a3b293cf489-kube-api-access-prj6g\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-var-lib-cni-bin\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-socket-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysconfig\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-hosts-file\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysconfig\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51c8b78-a028-4ace-8861-369f61795c3b-konnectivity-ca\") pod \"konnectivity-agent-h6dbd\" (UID: \"b51c8b78-a028-4ace-8861-369f61795c3b\") " pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-socket-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-os-release\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-os-release\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-sys-fs\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.349821 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.349011 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-sys-fs\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.348974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4fm\" (UniqueName: \"kubernetes.io/projected/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-kube-api-access-fc4fm\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.349106 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:36.849068698 +0000 UTC m=+1.995884937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-run-netns\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-cni-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-cnibin\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-cni-dir\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppzl\" (UniqueName: \"kubernetes.io/projected/195010ea-cabb-4497-9b46-c46ab80c6137-kube-api-access-lppzl\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-cnibin\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-systemd\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-cni-netd\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-etc-kubernetes\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.350625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7vf\" (UniqueName: \"kubernetes.io/projected/fc8f68d2-8865-437d-b247-afaac34a9474-kube-api-access-bp7vf\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-etc-kubernetes\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-var-lib-kubelet\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-systemd-units\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44dd09e6-1d0f-4ea6-8637-4531305f5c46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-node-log\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2d36207-38b9-4ace-a64d-f0439dea7bdc-kubelet-config\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-var-lib-kubelet\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-ovnkube-script-lib\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-socket-dir-parent\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-serviceca\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-system-cni-dir\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-slash\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdxs\" (UniqueName: \"kubernetes.io/projected/098d0be6-e403-4144-ab99-3835833be835-kube-api-access-fpdxs\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-multus-socket-dir-parent\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.351230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-kubernetes\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-systemd\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cni-binary-copy\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.349999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-var-lib-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-env-overrides\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-systemd\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysctl-d\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-host\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-netns\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-sysctl-d\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc8f68d2-8865-437d-b247-afaac34a9474-host-run-netns\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-etc-kubernetes\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.350271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/195010ea-cabb-4497-9b46-c46ab80c6137-host\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.351498 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/195010ea-cabb-4497-9b46-c46ab80c6137-etc-tuned\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.351808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51c8b78-a028-4ace-8861-369f61795c3b-agent-certs\") pod \"konnectivity-agent-h6dbd\" (UID: \"b51c8b78-a028-4ace-8861-369f61795c3b\") " pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.352062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.351977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/195010ea-cabb-4497-9b46-c46ab80c6137-tmp\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.357818 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.357796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4gz\" (UniqueName: \"kubernetes.io/projected/44dd09e6-1d0f-4ea6-8637-4531305f5c46-kube-api-access-nm4gz\") pod \"aws-ebs-csi-driver-node-l6vd2\" (UID: \"44dd09e6-1d0f-4ea6-8637-4531305f5c46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.361763 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.361737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prj6g\" (UniqueName: \"kubernetes.io/projected/9a17151a-4a1c-4988-95f7-8a3b293cf489-kube-api-access-prj6g\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.363479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.363457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppzl\" (UniqueName: \"kubernetes.io/projected/195010ea-cabb-4497-9b46-c46ab80c6137-kube-api-access-lppzl\") pod \"tuned-8x5kr\" (UID: \"195010ea-cabb-4497-9b46-c46ab80c6137\") " pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.363894 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.363873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7vf\" (UniqueName: \"kubernetes.io/projected/fc8f68d2-8865-437d-b247-afaac34a9474-kube-api-access-bp7vf\") pod \"multus-w7w54\" (UID: \"fc8f68d2-8865-437d-b247-afaac34a9474\") " pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.405378 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.405333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90563c7e7eb521e876399cd564ee59a0.slice/crio-49093b666578a5fe31dba734ba69c10e9e0209c68f029c8ec568a5cae827bd2d WatchSource:0}: Error finding container 49093b666578a5fe31dba734ba69c10e9e0209c68f029c8ec568a5cae827bd2d: Status 404 returned error can't find the container with id 49093b666578a5fe31dba734ba69c10e9e0209c68f029c8ec568a5cae827bd2d Aug 04 18:10:36.405803 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.405776 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9c4aecedc139bae6541d09b9bc2bed.slice/crio-f72cdf62f2e2ccf5b8cbc18dec46ad8e621d377807a6808bc38e89f4cbccd3a2 WatchSource:0}: Error finding container f72cdf62f2e2ccf5b8cbc18dec46ad8e621d377807a6808bc38e89f4cbccd3a2: Status 404 returned error can't find the container with id f72cdf62f2e2ccf5b8cbc18dec46ad8e621d377807a6808bc38e89f4cbccd3a2 Aug 04 18:10:36.410593 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.410576 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:10:36.417210 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.417187 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:36.450537 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-run-netns\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450537 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-systemd\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-cni-netd\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-run-netns\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-systemd\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-cni-netd\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450661 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-systemd-units\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-systemd-units\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-node-log\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2d36207-38b9-4ace-a64d-f0439dea7bdc-kubelet-config\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-ovnkube-script-lib\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.450765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-serviceca\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-node-log\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-system-cni-dir\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-system-cni-dir\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450837 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2d36207-38b9-4ace-a64d-f0439dea7bdc-kubelet-config\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-slash\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450893 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-slash\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdxs\" (UniqueName: \"kubernetes.io/projected/098d0be6-e403-4144-ab99-3835833be835-kube-api-access-fpdxs\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cni-binary-copy\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.450991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-var-lib-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-env-overrides\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-var-lib-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-cni-bin\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-ovnkube-config\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqk9\" (UniqueName: \"kubernetes.io/projected/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-kube-api-access-4jqk9\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-cni-bin\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.451381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cnibin\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-etc-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-serviceca\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/098d0be6-e403-4144-ab99-3835833be835-ovn-node-metrics-cert\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-etc-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cni-binary-copy\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cnibin\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-tmp-dir\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-ovnkube-script-lib\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-env-overrides\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.451583 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-host-slash\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.451766 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:10:36.951739681 +0000 UTC m=+2.098555926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-host-slash\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.452185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451783 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/098d0be6-e403-4144-ab99-3835833be835-ovnkube-config\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-iptables-alerter-script\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-openvswitch\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-kubelet\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-ovn\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-log-socket\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-tmp-dir\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2d36207-38b9-4ace-a64d-f0439dea7bdc-dbus\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-run-ovn\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-host\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.451999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw624\" (UniqueName: \"kubernetes.io/projected/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-kube-api-access-gw624\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-host\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-os-release\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6jd\" (UniqueName: \"kubernetes.io/projected/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-kube-api-access-hk6jd\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452085 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-os-release\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452086 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.452960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2d36207-38b9-4ace-a64d-f0439dea7bdc-dbus\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-kubelet\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-hosts-file\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-log-socket\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/098d0be6-e403-4144-ab99-3835833be835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-hosts-file\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4fm\" (UniqueName: \"kubernetes.io/projected/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-kube-api-access-fc4fm\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-iptables-alerter-script\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.453579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.452576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.454155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.454137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/098d0be6-e403-4144-ab99-3835833be835-ovn-node-metrics-cert\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.461785 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.461763 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:36.461785 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.461784 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:36.461975 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.461795 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:36.461975 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.461847 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:10:36.96183279 +0000 UTC m=+2.108649029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:36.465570 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.465533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw624\" (UniqueName: \"kubernetes.io/projected/b4763dc2-4f1b-4be6-a14b-0bdb04fd9206-kube-api-access-gw624\") pod \"node-ca-xn6wp\" (UID: \"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206\") " pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.465688 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.465573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4fm\" (UniqueName: \"kubernetes.io/projected/cbb59dbb-5a64-443d-b7dd-b6a66d663e85-kube-api-access-fc4fm\") pod \"iptables-alerter-2w24j\" (UID: \"cbb59dbb-5a64-443d-b7dd-b6a66d663e85\") " pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.465768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.465750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdxs\" (UniqueName: \"kubernetes.io/projected/098d0be6-e403-4144-ab99-3835833be835-kube-api-access-fpdxs\") pod \"ovnkube-node-pcjf4\" (UID: \"098d0be6-e403-4144-ab99-3835833be835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.466132 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.466105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqk9\" (UniqueName: \"kubernetes.io/projected/5f8c2fcb-e227-4861-8e7a-5152a4ed3d24-kube-api-access-4jqk9\") pod \"node-resolver-tjjg9\" (UID: \"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24\") " pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.466230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.466170 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6jd\" (UniqueName: \"kubernetes.io/projected/286e235a-d4c1-4bf7-aa18-e2b7704b6d68-kube-api-access-hk6jd\") pod \"multus-additional-cni-plugins-c896x\" (UID: \"286e235a-d4c1-4bf7-aa18-e2b7704b6d68\") " pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.559340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.559256 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7w54" Aug 04 18:10:36.570634 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.570598 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8f68d2_8865_437d_b247_afaac34a9474.slice/crio-aa6c6dcd8777015620f07434a17b259779c36d0eeffe94f26e2e5a4504f77059 WatchSource:0}: Error finding container aa6c6dcd8777015620f07434a17b259779c36d0eeffe94f26e2e5a4504f77059: Status 404 returned error can't find the container with id aa6c6dcd8777015620f07434a17b259779c36d0eeffe94f26e2e5a4504f77059 Aug 04 18:10:36.572727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.572696 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:10:36.581584 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.581559 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51c8b78_a028_4ace_8861_369f61795c3b.slice/crio-1f67660584f35cf62d9412fddf0e1299b914b37345e4a4307c9a95e0724eb9a9 WatchSource:0}: Error finding container 1f67660584f35cf62d9412fddf0e1299b914b37345e4a4307c9a95e0724eb9a9: Status 404 returned error can't find the container with id 1f67660584f35cf62d9412fddf0e1299b914b37345e4a4307c9a95e0724eb9a9 Aug 04 18:10:36.584691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.584673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" Aug 04 18:10:36.594187 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.592831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" Aug 04 18:10:36.594455 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.594411 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dd09e6_1d0f_4ea6_8637_4531305f5c46.slice/crio-f0949ec784c7c01a3d9d174722ea070e0925d7491936272512cd7f36b9f43886 WatchSource:0}: Error finding container f0949ec784c7c01a3d9d174722ea070e0925d7491936272512cd7f36b9f43886: Status 404 returned error can't find the container with id f0949ec784c7c01a3d9d174722ea070e0925d7491936272512cd7f36b9f43886 Aug 04 18:10:36.602499 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.602474 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195010ea_cabb_4497_9b46_c46ab80c6137.slice/crio-1b2e8b9a99cbedec52cfee1cb2dafd532f87ae5627b597f7129211ebf97067e9 WatchSource:0}: Error finding container 1b2e8b9a99cbedec52cfee1cb2dafd532f87ae5627b597f7129211ebf97067e9: Status 404 returned error can't find the container with id 1b2e8b9a99cbedec52cfee1cb2dafd532f87ae5627b597f7129211ebf97067e9 Aug 04 18:10:36.607367 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.607345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjjg9" Aug 04 18:10:36.615950 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.615924 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8c2fcb_e227_4861_8e7a_5152a4ed3d24.slice/crio-727166929f59c1bbd44c11430cd4bd39ebbd2a8aa1636cb9e426877b4d7a9cbd WatchSource:0}: Error finding container 727166929f59c1bbd44c11430cd4bd39ebbd2a8aa1636cb9e426877b4d7a9cbd: Status 404 returned error can't find the container with id 727166929f59c1bbd44c11430cd4bd39ebbd2a8aa1636cb9e426877b4d7a9cbd Aug 04 18:10:36.623307 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.623285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xn6wp" Aug 04 18:10:36.631700 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.631677 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4763dc2_4f1b_4be6_a14b_0bdb04fd9206.slice/crio-b58f69ec07ee3825b6e01dcbf7bb1327ea6ca93e89c86f6fab1b0ba6cf30625c WatchSource:0}: Error finding container b58f69ec07ee3825b6e01dcbf7bb1327ea6ca93e89c86f6fab1b0ba6cf30625c: Status 404 returned error can't find the container with id b58f69ec07ee3825b6e01dcbf7bb1327ea6ca93e89c86f6fab1b0ba6cf30625c Aug 04 18:10:36.641234 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.641214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2w24j" Aug 04 18:10:36.642973 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.642951 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:36.650213 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.650188 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb59dbb_5a64_443d_b7dd_b6a66d663e85.slice/crio-fed04cf669a27834e545b0c4f094ce229c1c605ce7eb6ff1b00d958528fa2c99 WatchSource:0}: Error finding container fed04cf669a27834e545b0c4f094ce229c1c605ce7eb6ff1b00d958528fa2c99: Status 404 returned error can't find the container with id fed04cf669a27834e545b0c4f094ce229c1c605ce7eb6ff1b00d958528fa2c99 Aug 04 18:10:36.666483 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.666463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c896x" Aug 04 18:10:36.670229 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.670211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:10:36.676898 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.676875 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286e235a_d4c1_4bf7_aa18_e2b7704b6d68.slice/crio-d8ab4504b89e111b8e921d2571f4816bf3b05769928a99676d7153ea49ce21c9 WatchSource:0}: Error finding container d8ab4504b89e111b8e921d2571f4816bf3b05769928a99676d7153ea49ce21c9: Status 404 returned error can't find the container with id d8ab4504b89e111b8e921d2571f4816bf3b05769928a99676d7153ea49ce21c9 Aug 04 18:10:36.680285 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:10:36.680265 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098d0be6_e403_4144_ab99_3835833be835.slice/crio-8c70302810be797c93a7505c0f83454fce9c1c1dfb4540965a6e83b8190d0d20 WatchSource:0}: Error finding container 8c70302810be797c93a7505c0f83454fce9c1c1dfb4540965a6e83b8190d0d20: Status 404 returned error can't find the container with id 8c70302810be797c93a7505c0f83454fce9c1c1dfb4540965a6e83b8190d0d20 Aug 04 18:10:36.854474 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.854357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:36.854623 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.854527 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:36.854623 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.854596 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:37.854578887 +0000 UTC m=+3.001395127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:36.954967 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:36.954927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:36.955186 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.955049 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:36.955186 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:36.955109 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:10:37.955094075 +0000 UTC m=+3.101910314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:37.056148 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.056111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:37.056374 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.056286 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:37.056374 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.056309 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:37.056374 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.056319 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:37.056374 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.056384 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:10:38.056363471 +0000 UTC m=+3.203179732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:37.218824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.218748 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 18:10:37.316074 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.316027 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 18:05:36 +0000 UTC" deadline="2028-05-09 18:42:46.418969894 +0000 UTC" Aug 04 18:10:37.316074 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.316073 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15456h32m9.102902323s" Aug 04 18:10:37.350553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.350492 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerStarted","Data":"d8ab4504b89e111b8e921d2571f4816bf3b05769928a99676d7153ea49ce21c9"} Aug 04 18:10:37.356307 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.356269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xn6wp" event={"ID":"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206","Type":"ContainerStarted","Data":"b58f69ec07ee3825b6e01dcbf7bb1327ea6ca93e89c86f6fab1b0ba6cf30625c"} Aug 04 18:10:37.358661 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.358630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7w54" event={"ID":"fc8f68d2-8865-437d-b247-afaac34a9474","Type":"ContainerStarted","Data":"aa6c6dcd8777015620f07434a17b259779c36d0eeffe94f26e2e5a4504f77059"} Aug 04 18:10:37.365195 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.365161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" event={"ID":"90563c7e7eb521e876399cd564ee59a0","Type":"ContainerStarted","Data":"49093b666578a5fe31dba734ba69c10e9e0209c68f029c8ec568a5cae827bd2d"} Aug 04 18:10:37.370286 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.369856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2w24j" event={"ID":"cbb59dbb-5a64-443d-b7dd-b6a66d663e85","Type":"ContainerStarted","Data":"fed04cf669a27834e545b0c4f094ce229c1c605ce7eb6ff1b00d958528fa2c99"} Aug 04 18:10:37.373181 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.373118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjjg9" event={"ID":"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24","Type":"ContainerStarted","Data":"727166929f59c1bbd44c11430cd4bd39ebbd2a8aa1636cb9e426877b4d7a9cbd"} Aug 04 18:10:37.375298 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.375255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" event={"ID":"195010ea-cabb-4497-9b46-c46ab80c6137","Type":"ContainerStarted","Data":"1b2e8b9a99cbedec52cfee1cb2dafd532f87ae5627b597f7129211ebf97067e9"} Aug 04 18:10:37.386568 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.386539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" event={"ID":"44dd09e6-1d0f-4ea6-8637-4531305f5c46","Type":"ContainerStarted","Data":"f0949ec784c7c01a3d9d174722ea070e0925d7491936272512cd7f36b9f43886"} Aug 04 18:10:37.390955 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.390897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6dbd" event={"ID":"b51c8b78-a028-4ace-8861-369f61795c3b","Type":"ContainerStarted","Data":"1f67660584f35cf62d9412fddf0e1299b914b37345e4a4307c9a95e0724eb9a9"} Aug 04 18:10:37.394041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.393990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" event={"ID":"aa9c4aecedc139bae6541d09b9bc2bed","Type":"ContainerStarted","Data":"f72cdf62f2e2ccf5b8cbc18dec46ad8e621d377807a6808bc38e89f4cbccd3a2"} Aug 04 18:10:37.396291 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.396264 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"8c70302810be797c93a7505c0f83454fce9c1c1dfb4540965a6e83b8190d0d20"} Aug 04 18:10:37.864491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.864451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:37.864693 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.864612 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:37.864765 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.864703 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:39.864678456 +0000 UTC m=+5.011494699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:37.965912 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:37.965870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:37.966127 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.966078 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:37.966192 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:37.966158 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:10:39.966127642 +0000 UTC m=+5.112943886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:38.066846 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:38.066804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:38.067055 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.067000 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:38.067055 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.067024 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:38.067055 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.067043 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:38.067227 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.067105 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:10:40.067086213 +0000 UTC m=+5.213902459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:38.316848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:38.316722 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 18:05:36 +0000 UTC" deadline="2028-02-02 17:45:26.394241639 +0000 UTC" Aug 04 18:10:38.316848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:38.316796 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13127h34m48.077450084s" Aug 04 18:10:38.342864 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:38.342831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:38.343070 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.342964 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:38.343461 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:38.343415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:38.343569 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.343553 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:38.343629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:38.343618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:38.343712 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:38.343693 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:39.883659 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:39.883332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:39.884174 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:39.883590 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:39.884174 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:39.883836 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:43.883813184 +0000 UTC m=+9.030629429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:39.984782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:39.984058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:39.984782 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:39.984248 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:39.984782 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:39.984317 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:10:43.984296545 +0000 UTC m=+9.131112787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:40.085020 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:40.084979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:40.085234 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.085152 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:40.085234 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.085172 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:40.085234 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.085185 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:40.085525 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.085252 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:10:44.085229147 +0000 UTC m=+9.232045409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:40.342731 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:40.342639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:40.342914 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.342806 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:40.343245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:40.343223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:40.343363 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.343337 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:40.343463 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:40.343396 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:40.343524 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:40.343490 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:42.343286 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:42.343244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:42.343286 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:42.343283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:42.343850 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:42.343283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:42.343850 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:42.343414 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:42.343850 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:42.343540 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:42.343850 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:42.343616 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:43.916958 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:43.916913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:43.917450 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:43.917053 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:43.917450 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:43.917133 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:10:51.917110508 +0000 UTC m=+17.063926755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:44.018005 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:44.017966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:44.018230 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.018197 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:44.018312 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.018290 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:10:52.018253349 +0000 UTC m=+17.165069598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:44.118966 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:44.118935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:44.119137 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.119055 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:44.119137 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.119071 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:44.119137 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.119080 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:44.119137 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.119125 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:10:52.119111763 +0000 UTC m=+17.265928000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:44.343181 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:44.343102 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:44.343345 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.343247 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:44.343559 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:44.343534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:44.343559 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:44.343564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:44.343734 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.343636 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:44.343734 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:44.343721 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:46.343051 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:46.343014 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:46.343562 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:46.343145 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:46.343562 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:46.343220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:46.343562 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:46.343308 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:46.343562 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:46.343356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:46.343562 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:46.343460 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:48.343276 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:48.343240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:48.343735 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:48.343401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:48.343735 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:48.343447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:48.343735 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:48.343517 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:48.343735 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:48.343573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:48.343735 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:48.343656 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:50.342808 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:50.342767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:50.343266 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:50.342767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:50.343266 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:50.342884 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:50.343266 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:50.342982 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:50.343266 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:50.342780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:50.343266 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:50.343088 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:51.977829 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:51.977792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:51.978284 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:51.977910 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:51.978284 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:51.977968 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:07.977948865 +0000 UTC m=+33.124765103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:10:52.079004 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:52.078971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:52.079178 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.079105 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:52.079178 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.079160 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:11:08.079146609 +0000 UTC m=+33.225962847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:10:52.180194 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:52.180156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:52.180388 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.180309 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:10:52.180388 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.180327 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:10:52.180388 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.180340 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:52.180540 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.180412 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:11:08.180396014 +0000 UTC m=+33.327212256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:10:52.343401 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:52.343296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:52.343401 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:52.343351 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:52.343401 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:52.343298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:52.343707 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.343461 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:52.343707 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.343530 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:52.343707 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:52.343607 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:54.342943 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:54.342912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:54.343435 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:54.342910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:54.343435 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:54.343024 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:54.343435 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:54.343084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:54.343435 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:54.343187 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:54.343435 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:54.343255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:56.343211 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:56.343182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:56.343686 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:56.343221 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:56.343686 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:56.343220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:56.343686 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:56.343307 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:56.343686 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:56.343399 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:56.343686 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:56.343447 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:58.343506 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.343175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:10:58.344353 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.343193 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:10:58.344353 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:58.343594 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:10:58.344353 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.343226 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:10:58.344353 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:58.343651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:10:58.344353 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:10:58.343730 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:10:58.488174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.488132 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xn6wp" event={"ID":"b4763dc2-4f1b-4be6-a14b-0bdb04fd9206","Type":"ContainerStarted","Data":"74de87a1d5d62752ba16ee7696ffd4b7408290a018dfbadb28022bc18435155a"} Aug 04 18:10:58.489754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.489720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7w54" event={"ID":"fc8f68d2-8865-437d-b247-afaac34a9474","Type":"ContainerStarted","Data":"98d7b29c8a7ceb7a26056bc2ab66e5d95631044943f9a6fd9902f94a0ab4b6f0"} Aug 04 18:10:58.491067 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.491042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" event={"ID":"90563c7e7eb521e876399cd564ee59a0","Type":"ContainerStarted","Data":"a10728453383f9b069ac1e5fceda9fdfc6693368306cdb824fb7c10ad1ea7bf5"} Aug 04 18:10:58.492497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.492476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjjg9" event={"ID":"5f8c2fcb-e227-4861-8e7a-5152a4ed3d24","Type":"ContainerStarted","Data":"e4d08659f6c29a9537b1a9e98f7b0a3f326f0ffdfd13592b9902b05d793844fd"} Aug 04 18:10:58.493710 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.493691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" event={"ID":"195010ea-cabb-4497-9b46-c46ab80c6137","Type":"ContainerStarted","Data":"5818436fb6a3c70b44d0e5958cc48f19c58c1c7f55db91fb5170aac71d2c1d5c"} Aug 04 18:10:58.495020 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.495003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" event={"ID":"44dd09e6-1d0f-4ea6-8637-4531305f5c46","Type":"ContainerStarted","Data":"babcf106a26e677ea95176ad05412b83dae84f2f6421cbe6585929beb48dd10d"} Aug 04 18:10:58.496123 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.496102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h6dbd" event={"ID":"b51c8b78-a028-4ace-8861-369f61795c3b","Type":"ContainerStarted","Data":"cf8c6ce692f8701c2a7a78310ead04357b83b2bd9430b3e0899e4383c3f22176"} Aug 04 18:10:58.497471 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.497447 2575 generic.go:356] "Generic (PLEG): container finished" podID="aa9c4aecedc139bae6541d09b9bc2bed" containerID="46bd42423959ac0256c467eac3dcc3d2640ed38aad5d25df333f4f9820acc0cb" exitCode=0 Aug 04 18:10:58.497552 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.497519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" event={"ID":"aa9c4aecedc139bae6541d09b9bc2bed","Type":"ContainerDied","Data":"46bd42423959ac0256c467eac3dcc3d2640ed38aad5d25df333f4f9820acc0cb"} Aug 04 18:10:58.500470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.500450 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"1ea4cbebcc44f5c511ce4ae92da82cee0d79251e81f01d67e4463f241420e854"} Aug 04 18:10:58.500566 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.500538 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"dc5c5de6b9936ffd78a2b9bcd2183e28f0ad1e2f3fc0b58afe7ae26f85f05a2e"} Aug 04 18:10:58.500609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.500579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"30db358fafd90c29aa25743aa28d378578c4598393b369d74cac08642e7da0e0"} Aug 04 18:10:58.500650 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.500614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"15a46eebe19f71ff8d2ffd37e0f0d3fe07354fed3010d5178760263a4bc8a379"} Aug 04 18:10:58.500686 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.500649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"4d549acdc499dbed026c28ce702278b1e67681b59bd60154213dc8a93211b6a5"} Aug 04 18:10:58.500720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.500694 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"83d923af04ad3cae15cbb8759db8544210adbed195aee6c26e3879bec13dfd54"} Aug 04 18:10:58.501737 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.501716 2575 generic.go:356] "Generic (PLEG): container finished" podID="286e235a-d4c1-4bf7-aa18-e2b7704b6d68" containerID="aec12591318a9b3c5fa0e7e08b0cd93cfe17af51fb5ad9c81623d84064bd89e8" exitCode=0 Aug 04 18:10:58.501848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.501745 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerDied","Data":"aec12591318a9b3c5fa0e7e08b0cd93cfe17af51fb5ad9c81623d84064bd89e8"} Aug 04 18:10:58.505470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.505430 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-222.ec2.internal" podStartSLOduration=22.505414771 podStartE2EDuration="22.505414771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:57.502846432 +0000 UTC m=+22.649662692" watchObservedRunningTime="2026-08-04 18:10:58.505414771 +0000 UTC m=+23.652231031" Aug 04 18:10:58.505804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.505781 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xn6wp" podStartSLOduration=1.834664503 podStartE2EDuration="22.505772711s" totalImagesPullingTime="20.671108208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:58.50502578 +0000 UTC m=+23.651842039" watchObservedRunningTime="2026-08-04 18:10:58.505772711 +0000 UTC m=+23.652588973" Aug 04 18:10:58.524450 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.524405 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8x5kr" podStartSLOduration=1.8220837859999999 podStartE2EDuration="22.524388505s" totalImagesPullingTime="20.702304719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:58.524123038 +0000 UTC m=+23.670939309" watchObservedRunningTime="2026-08-04 18:10:58.524388505 +0000 UTC m=+23.671204765" Aug 04 18:10:58.593492 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.593448 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w7w54" podStartSLOduration=2.8057834809999997 podStartE2EDuration="23.593432948s" totalImagesPullingTime="20.787649467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:58.593239996 +0000 UTC m=+23.740056257" watchObservedRunningTime="2026-08-04 18:10:58.593432948 +0000 UTC m=+23.740249207" Aug 04 18:10:58.614291 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.614239 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h6dbd" podStartSLOduration=3.201725788 podStartE2EDuration="23.614222667s" totalImagesPullingTime="20.412496879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:58.614199181 +0000 UTC m=+23.761015440" watchObservedRunningTime="2026-08-04 18:10:58.614222667 +0000 UTC m=+23.761038926" Aug 04 18:10:58.632240 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:58.632197 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tjjg9" podStartSLOduration=1.93888167 podStartE2EDuration="22.632181564s" totalImagesPullingTime="20.693299894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:58.631704552 +0000 UTC m=+23.778520812" watchObservedRunningTime="2026-08-04 18:10:58.632181564 +0000 UTC m=+23.778997823" Aug 04 18:10:59.506110 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:59.506066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2w24j" event={"ID":"cbb59dbb-5a64-443d-b7dd-b6a66d663e85","Type":"ContainerStarted","Data":"038272b0c6cb249141f3977629f9ea2650f73005efd6b9924e6c4b9601eb1064"} Aug 04 18:10:59.508625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:59.508500 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" event={"ID":"aa9c4aecedc139bae6541d09b9bc2bed","Type":"ContainerStarted","Data":"27977753c5c5ec84740670700d1a315fea964f65071a96e774dfe96423049a9d"} Aug 04 18:10:59.525350 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:59.525288 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2w24j" podStartSLOduration=2.868613998 podStartE2EDuration="23.525265258s" totalImagesPullingTime="20.65665126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:59.524453861 +0000 UTC m=+24.671270122" watchObservedRunningTime="2026-08-04 18:10:59.525265258 +0000 UTC m=+24.672081519" Aug 04 18:10:59.541568 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:59.541512 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-222.ec2.internal" podStartSLOduration=23.541497671 podStartE2EDuration="23.541497671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:59.541178794 +0000 UTC m=+24.687995055" watchObservedRunningTime="2026-08-04 18:10:59.541497671 +0000 UTC m=+24.688313932" Aug 04 18:10:59.868064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:10:59.868038 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 18:11:00.315247 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.315143 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:10:59.868056989Z","UUID":"92e6a9f6-c82e-40e3-a544-e3c1151b6479","Handler":null,"Name":"","Endpoint":""} Aug 04 18:11:00.317249 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.317221 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 18:11:00.317386 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.317264 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 18:11:00.342948 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.342923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:00.343056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.342959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:00.343056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.342923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:00.343056 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:00.343033 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:11:00.343201 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:00.343157 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:11:00.343274 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:00.343244 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:11:00.514800 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.514759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"3fcccfced90b19f396feb3adb4aa9f9f4fd35045f566c61718ee6414d4c56b35"} Aug 04 18:11:00.516783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:00.516741 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" event={"ID":"44dd09e6-1d0f-4ea6-8637-4531305f5c46","Type":"ContainerStarted","Data":"540aa5ac7380d06fd9c0a0d186d0d2a13cbda758086c5249f21dd4d75ff4fe2c"} Aug 04 18:11:01.521011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:01.520928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" event={"ID":"44dd09e6-1d0f-4ea6-8637-4531305f5c46","Type":"ContainerStarted","Data":"29210525be69052c2982a2d86c1ba83307098d7e36b2804090f55989109504e7"} Aug 04 18:11:01.544170 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:01.544114 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vd2" podStartSLOduration=2.105750428 podStartE2EDuration="25.54409888s" totalImagesPullingTime="23.438348452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:01.543172464 +0000 UTC m=+26.689988723" watchObservedRunningTime="2026-08-04 18:11:01.54409888 +0000 UTC m=+26.690915173" Aug 04 18:11:01.973854 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:01.973775 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:11:01.974479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:01.974456 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:11:02.342804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:02.342725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:02.342804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:02.342744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:02.342999 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:02.342735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:02.342999 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:02.342840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:11:02.342999 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:02.342953 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:11:02.343125 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:02.343036 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:11:02.523646 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:02.523579 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:11:02.524099 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:02.523985 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h6dbd" Aug 04 18:11:04.342879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.342672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:04.343619 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.342672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:04.343619 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:04.342961 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:11:04.343619 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.342672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:04.343619 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:04.343031 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:11:04.343619 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:04.343096 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:11:04.530105 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.530061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" event={"ID":"098d0be6-e403-4144-ab99-3835833be835","Type":"ContainerStarted","Data":"9b5ea3bf8f9c03a5ae15f9f8399df0098d8816a509c7c7175ed9681970065dd0"} Aug 04 18:11:04.530417 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.530312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:11:04.530417 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.530409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:11:04.532185 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.532155 2575 generic.go:356] "Generic (PLEG): container finished" podID="286e235a-d4c1-4bf7-aa18-e2b7704b6d68" containerID="49c755410a1b9463e94e1b9def70fca09ec8a28dd0cdc783cff44d36c5ac9ce0" exitCode=0 Aug 04 18:11:04.532306 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.532218 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerDied","Data":"49c755410a1b9463e94e1b9def70fca09ec8a28dd0cdc783cff44d36c5ac9ce0"} Aug 04 18:11:04.546959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.546933 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:11:04.563571 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:04.563514 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" podStartSLOduration=7.8711725569999995 podStartE2EDuration="28.563499535s" totalImagesPullingTime="20.692326978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:04.562355686 +0000 UTC m=+29.709171945" watchObservedRunningTime="2026-08-04 18:11:04.563499535 +0000 UTC m=+29.710315795" Aug 04 18:11:05.535475 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.535220 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:11:05.550782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.550758 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:11:05.861417 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.861283 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fkfc"] Aug 04 18:11:05.861544 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.861459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:05.861593 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:05.861575 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:11:05.865033 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.865006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zv8cr"] Aug 04 18:11:05.865163 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.865150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:05.865272 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:05.865252 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:11:05.865793 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.865770 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fwqwx"] Aug 04 18:11:05.865887 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:05.865874 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:05.865980 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:05.865945 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:11:06.539164 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:06.539128 2575 generic.go:356] "Generic (PLEG): container finished" podID="286e235a-d4c1-4bf7-aa18-e2b7704b6d68" containerID="71bf6619136892ddbf960e145554e1a5e7a64fb7dcb1160a3f702c5052c8e522" exitCode=0 Aug 04 18:11:06.539675 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:06.539216 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerDied","Data":"71bf6619136892ddbf960e145554e1a5e7a64fb7dcb1160a3f702c5052c8e522"} Aug 04 18:11:07.342727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:07.342639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:07.342727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:07.342684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:07.342941 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:07.342639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:07.342941 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:07.342774 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:11:07.342941 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:07.342872 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:11:07.343110 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:07.342974 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:11:08.001654 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:08.001620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:08.002090 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.001798 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:11:08.002090 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.001870 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs podName:9a17151a-4a1c-4988-95f7-8a3b293cf489 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:40.001853019 +0000 UTC m=+65.148669261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs") pod "network-metrics-daemon-zv8cr" (UID: "9a17151a-4a1c-4988-95f7-8a3b293cf489") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 18:11:08.102295 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:08.102250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:08.102519 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.102456 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 18:11:08.102589 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.102542 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret podName:c2d36207-38b9-4ace-a64d-f0439dea7bdc nodeName:}" failed. No retries permitted until 2026-08-04 18:11:40.102521025 +0000 UTC m=+65.249337274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret") pod "global-pull-secret-syncer-4fkfc" (UID: "c2d36207-38b9-4ace-a64d-f0439dea7bdc") : object "kube-system"/"original-pull-secret" not registered Aug 04 18:11:08.203643 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:08.203547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:08.203802 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.203698 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 18:11:08.203802 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.203720 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 18:11:08.203802 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.203731 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cp74s for pod openshift-network-diagnostics/network-check-target-fwqwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:11:08.203802 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:08.203792 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s podName:ff35f4cf-3ecb-4ac5-8132-28e836321b8d nodeName:}" failed. No retries permitted until 2026-08-04 18:11:40.203775871 +0000 UTC m=+65.350592109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp74s" (UniqueName: "kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s") pod "network-check-target-fwqwx" (UID: "ff35f4cf-3ecb-4ac5-8132-28e836321b8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 18:11:08.545106 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:08.545030 2575 generic.go:356] "Generic (PLEG): container finished" podID="286e235a-d4c1-4bf7-aa18-e2b7704b6d68" containerID="c1661b4ca1c657deb82c952e7847e5992eaad2431ea3c073f697cbe1ec5cf8df" exitCode=0 Aug 04 18:11:08.545106 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:08.545084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerDied","Data":"c1661b4ca1c657deb82c952e7847e5992eaad2431ea3c073f697cbe1ec5cf8df"} Aug 04 18:11:09.343290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:09.343258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:09.343290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:09.343285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:09.343831 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:09.343258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:09.343831 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:09.343411 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zv8cr" podUID="9a17151a-4a1c-4988-95f7-8a3b293cf489" Aug 04 18:11:09.343831 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:09.343506 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwqwx" podUID="ff35f4cf-3ecb-4ac5-8132-28e836321b8d" Aug 04 18:11:09.343831 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:09.343614 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fkfc" podUID="c2d36207-38b9-4ace-a64d-f0439dea7bdc" Aug 04 18:11:10.168676 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.168394 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-222.ec2.internal" event="NodeReady" Aug 04 18:11:10.168905 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.168793 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 18:11:10.223508 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.223476 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v"] Aug 04 18:11:10.228535 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.228509 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76fbdcb99f-qzxkm"] Aug 04 18:11:10.228694 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.228633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.230897 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.230867 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 18:11:10.231014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.230912 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 18:11:10.231014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.230954 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jpv7p\"" Aug 04 18:11:10.231014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.230993 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.231302 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.231289 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.232295 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.232276 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69fbf496bb-v6nzt"] Aug 04 18:11:10.232430 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.232416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.234601 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.234570 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 18:11:10.234698 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.234609 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg4n\"" Aug 04 18:11:10.234867 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.234853 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 18:11:10.234918 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.234865 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 18:11:10.236142 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.235875 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49"] Aug 04 18:11:10.236142 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.235999 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.240432 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.239387 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.240432 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.239761 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 18:11:10.240582 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.240439 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 18:11:10.244530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.240910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.244530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.241033 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-b6rd5\"" Aug 04 18:11:10.244530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.241105 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 18:11:10.244530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.242273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.245465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.244912 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 18:11:10.245465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.245127 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v"] Aug 04 18:11:10.246726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.246704 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49"] Aug 04 18:11:10.248046 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.247996 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.248328 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.248310 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.248522 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.248502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69fbf496bb-v6nzt"] Aug 04 18:11:10.248604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.248573 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm"] Aug 04 18:11:10.249041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.249023 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-b7zzs\"" Aug 04 18:11:10.249523 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.249503 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 18:11:10.249767 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.249747 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 18:11:10.250647 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.250498 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 18:11:10.252638 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.252613 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-m7zkx"] Aug 04 18:11:10.253205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.253182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" Aug 04 18:11:10.255983 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.255965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.259114 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.259094 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.259799 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.259778 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nw6nz\"" Aug 04 18:11:10.261180 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.261154 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.268644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.268626 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 18:11:10.269788 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.269768 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2lgls\"" Aug 04 18:11:10.270073 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.269991 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.274971 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.274949 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.274971 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.274962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 18:11:10.278029 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.278009 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76fbdcb99f-qzxkm"] Aug 04 18:11:10.278184 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.278164 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm"] Aug 04 18:11:10.284258 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.284239 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-m7zkx"] Aug 04 18:11:10.286597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.286578 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5"] Aug 04 18:11:10.290491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.290468 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x"] Aug 04 18:11:10.290607 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.290593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:10.293765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.293742 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-8hh79"] Aug 04 18:11:10.293880 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.293863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.296091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.296069 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 18:11:10.296803 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.296783 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-94wjp"] Aug 04 18:11:10.297032 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.297004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.300130 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.300104 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cvcw2"] Aug 04 18:11:10.302736 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.302704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.305066 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.305049 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.306302 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.306285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 18:11:10.316838 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.316811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-serving-cert\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.316951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.316848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.316951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.316876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkq7\" (UniqueName: \"kubernetes.io/projected/2363937d-6717-4808-85f9-de0fe414de74-kube-api-access-bnkq7\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.316951 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.316922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.317113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-stats-auth\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.317113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9jm\" (UniqueName: \"kubernetes.io/projected/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-kube-api-access-td9jm\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.317113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.317113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.317308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317147 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986cl\" (UniqueName: \"kubernetes.io/projected/40806ebc-9194-4588-93ba-7f9563faaf25-kube-api-access-986cl\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.317308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-certificates\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-snapshots\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.317308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-image-registry-private-configuration\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-default-certificate\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.317308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-bound-sa-token\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-service-ca-bundle\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4x7\" (UniqueName: \"kubernetes.io/projected/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-kube-api-access-6v4x7\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-trusted-ca\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-installation-pull-secrets\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-tmp\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2bc\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-kube-api-access-9n2bc\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldfz\" (UniqueName: \"kubernetes.io/projected/1fb0ddae-2f89-4c20-b7dc-12822928ddb6-kube-api-access-rldfz\") pod \"volume-data-source-validator-5b8776669-s7tzm\" (UID: \"1fb0ddae-2f89-4c20-b7dc-12822928ddb6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" Aug 04 18:11:10.317993 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-trusted-ca-bundle\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.317993 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.317993 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-ca-trust-extracted\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.317993 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.317697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2363937d-6717-4808-85f9-de0fe414de74-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.319614 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.319597 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.319748 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.319713 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.320349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.320181 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.320479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.320409 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zlqfr\"" Aug 04 18:11:10.320679 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.320624 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.321303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.320879 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.321303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.320888 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 18:11:10.321303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.321043 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.321303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.321076 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 18:11:10.321303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.321188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 18:11:10.321629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.321389 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-khq2z\"" Aug 04 18:11:10.321629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.321532 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9jwcr\"" Aug 04 18:11:10.322483 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.322466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-94wjp"] Aug 04 18:11:10.334462 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.334439 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 18:11:10.338985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.338968 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.340318 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.340301 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 18:11:10.341104 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.341083 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7cmc5\"" Aug 04 18:11:10.341231 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.341122 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 18:11:10.341231 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.341123 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 18:11:10.345028 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.345011 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.345503 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.345012 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwz87\"" Aug 04 18:11:10.350041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.350020 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 18:11:10.350529 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.350498 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-8hh79"] Aug 04 18:11:10.355683 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.355659 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5"] Aug 04 18:11:10.359108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.359084 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x"] Aug 04 18:11:10.379545 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.379469 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cvcw2"] Aug 04 18:11:10.380299 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.380278 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75"] Aug 04 18:11:10.383880 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.383857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" Aug 04 18:11:10.387147 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.387121 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dflk2\"" Aug 04 18:11:10.387424 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.387397 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 18:11:10.388464 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.388442 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 18:11:10.394134 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.394112 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75"] Aug 04 18:11:10.418857 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.418776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-service-ca-bundle\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.418857 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.418808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4x7\" (UniqueName: \"kubernetes.io/projected/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-kube-api-access-6v4x7\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.418857 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.418835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-installation-pull-secrets\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419084 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-tmp\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419130 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xz7\" (UniqueName: \"kubernetes.io/projected/dc3a9444-8ab5-4152-aadd-c700368fc215-kube-api-access-p9xz7\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.419164 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldfz\" (UniqueName: \"kubernetes.io/projected/1fb0ddae-2f89-4c20-b7dc-12822928ddb6-kube-api-access-rldfz\") pod \"volume-data-source-validator-5b8776669-s7tzm\" (UID: \"1fb0ddae-2f89-4c20-b7dc-12822928ddb6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" Aug 04 18:11:10.419164 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-ca-trust-extracted\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419232 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-trusted-ca-bundle\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419276 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.419465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-tmp\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419513 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc3a9444-8ab5-4152-aadd-c700368fc215-config\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.419546 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2363937d-6717-4808-85f9-de0fe414de74-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.419578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-service-ca-bundle\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a154ea0c-86e0-4615-8314-450a19397b28-serving-cert\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.419688 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-serving-cert\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419688 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkq7\" (UniqueName: \"kubernetes.io/projected/2363937d-6717-4808-85f9-de0fe414de74-kube-api-access-bnkq7\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.419688 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-bound-sa-token\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9jm\" (UniqueName: \"kubernetes.io/projected/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-kube-api-access-td9jm\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-certificates\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a154ea0c-86e0-4615-8314-450a19397b28-config\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.419804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55763a8-3b0c-4f5a-8e10-bc6964591a81-cert\") pod \"ingress-canary-94wjp\" (UID: \"b55763a8-3b0c-4f5a-8e10-bc6964591a81\") " pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xxp\" (UniqueName: \"kubernetes.io/projected/b55763a8-3b0c-4f5a-8e10-bc6964591a81-kube-api-access-z6xxp\") pod \"ingress-canary-94wjp\" (UID: \"b55763a8-3b0c-4f5a-8e10-bc6964591a81\") " pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-ca-trust-extracted\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-default-certificate\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4944n\" (UniqueName: \"kubernetes.io/projected/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-kube-api-access-4944n\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-trusted-ca\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2bc\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-kube-api-access-9n2bc\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-trusted-ca-bundle\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.419978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.419978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3935d47-1a3e-4d52-b644-62a4fe867dfb-tmp-dir\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3935d47-1a3e-4d52-b644-62a4fe867dfb-config-volume\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmgm\" (UniqueName: \"kubernetes.io/projected/b3935d47-1a3e-4d52-b644-62a4fe867dfb-kube-api-access-zhmgm\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrq5\" (UniqueName: \"kubernetes.io/projected/a154ea0c-86e0-4615-8314-450a19397b28-kube-api-access-whrq5\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420142 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-stats-auth\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420203 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:10.920181679 +0000 UTC m=+36.066997920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : secret "router-metrics-certs-default" not found Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2363937d-6717-4808-85f9-de0fe414de74-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420423 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-certificates\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.420468 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420479 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls podName:2363937d-6717-4808-85f9-de0fe414de74 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:10.920465959 +0000 UTC m=+36.067282216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-h5r49" (UID: "2363937d-6717-4808-85f9-de0fe414de74") : secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420493 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420510 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fbdcb99f-qzxkm: secret "image-registry-tls" not found Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc3a9444-8ab5-4152-aadd-c700368fc215-serving-cert\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a154ea0c-86e0-4615-8314-450a19397b28-trusted-ca\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.420793 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls podName:5af49ee1-5ad0-4f19-8765-0ae0c18acb08 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:10.920776678 +0000 UTC m=+36.067592916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls") pod "image-registry-76fbdcb99f-qzxkm" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08") : secret "image-registry-tls" not found Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.420980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.421056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.421022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986cl\" (UniqueName: \"kubernetes.io/projected/40806ebc-9194-4588-93ba-7f9563faaf25-kube-api-access-986cl\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.421563 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.421261 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:10.921148022 +0000 UTC m=+36.067964275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : configmap references non-existent config key: service-ca.crt Aug 04 18:11:10.421563 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.421284 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.421563 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.421303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-snapshots\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.421563 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.421440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-image-registry-private-configuration\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.421563 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.421462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-trusted-ca\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.421952 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.421928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-snapshots\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.423987 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.423956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.424111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.423995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-serving-cert\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.424111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.424004 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-image-registry-private-configuration\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.424111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.424092 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-default-certificate\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.424262 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.424203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-installation-pull-secrets\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.424435 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.424416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-stats-auth\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.435465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.435110 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkq7\" (UniqueName: \"kubernetes.io/projected/2363937d-6717-4808-85f9-de0fe414de74-kube-api-access-bnkq7\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.435465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.435118 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9jm\" (UniqueName: \"kubernetes.io/projected/b36af2a3-d7d0-469a-a78d-c20e9ffcb746-kube-api-access-td9jm\") pod \"insights-operator-77696685b-m7zkx\" (UID: \"b36af2a3-d7d0-469a-a78d-c20e9ffcb746\") " pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.435465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.435354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldfz\" (UniqueName: \"kubernetes.io/projected/1fb0ddae-2f89-4c20-b7dc-12822928ddb6-kube-api-access-rldfz\") pod \"volume-data-source-validator-5b8776669-s7tzm\" (UID: \"1fb0ddae-2f89-4c20-b7dc-12822928ddb6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" Aug 04 18:11:10.435725 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.435678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-986cl\" (UniqueName: \"kubernetes.io/projected/40806ebc-9194-4588-93ba-7f9563faaf25-kube-api-access-986cl\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.435897 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.435864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2bc\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-kube-api-access-9n2bc\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.436263 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.436221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4x7\" (UniqueName: \"kubernetes.io/projected/f8d6a004-71d5-46dc-b7bb-06bfa86f8131-kube-api-access-6v4x7\") pod \"kube-storage-version-migrator-operator-7878f4b88c-v666v\" (UID: \"f8d6a004-71d5-46dc-b7bb-06bfa86f8131\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.436708 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.436687 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-bound-sa-token\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.522578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xz7\" (UniqueName: \"kubernetes.io/projected/dc3a9444-8ab5-4152-aadd-c700368fc215-kube-api-access-p9xz7\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.522578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc3a9444-8ab5-4152-aadd-c700368fc215-config\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.522813 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a154ea0c-86e0-4615-8314-450a19397b28-serving-cert\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.522813 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9x4\" (UniqueName: \"kubernetes.io/projected/b6f687fc-950c-4152-8da1-ba0884fcd082-kube-api-access-ts9x4\") pod \"network-check-source-65bdbbff45-xkj75\" (UID: \"b6f687fc-950c-4152-8da1-ba0884fcd082\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" Aug 04 18:11:10.522937 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a154ea0c-86e0-4615-8314-450a19397b28-config\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.522937 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55763a8-3b0c-4f5a-8e10-bc6964591a81-cert\") pod \"ingress-canary-94wjp\" (UID: \"b55763a8-3b0c-4f5a-8e10-bc6964591a81\") " pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.522937 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xxp\" (UniqueName: \"kubernetes.io/projected/b55763a8-3b0c-4f5a-8e10-bc6964591a81-kube-api-access-z6xxp\") pod \"ingress-canary-94wjp\" (UID: \"b55763a8-3b0c-4f5a-8e10-bc6964591a81\") " pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.522937 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4944n\" (UniqueName: \"kubernetes.io/projected/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-kube-api-access-4944n\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:10.523137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.522959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3935d47-1a3e-4d52-b644-62a4fe867dfb-tmp-dir\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.523137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3935d47-1a3e-4d52-b644-62a4fe867dfb-config-volume\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.523137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmgm\" (UniqueName: \"kubernetes.io/projected/b3935d47-1a3e-4d52-b644-62a4fe867dfb-kube-api-access-zhmgm\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.523137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.523137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrq5\" (UniqueName: \"kubernetes.io/projected/a154ea0c-86e0-4615-8314-450a19397b28-kube-api-access-whrq5\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.523403 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:10.523403 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc3a9444-8ab5-4152-aadd-c700368fc215-serving-cert\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.523403 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.523186 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:10.523403 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.523256 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls podName:b3935d47-1a3e-4d52-b644-62a4fe867dfb nodeName:}" failed. No retries permitted until 2026-08-04 18:11:11.023233337 +0000 UTC m=+36.170049577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls") pod "dns-default-cvcw2" (UID: "b3935d47-1a3e-4d52-b644-62a4fe867dfb") : secret "dns-default-metrics-tls" not found Aug 04 18:11:10.523609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a154ea0c-86e0-4615-8314-450a19397b28-trusted-ca\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.523609 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.523485 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 18:11:10.523713 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3935d47-1a3e-4d52-b644-62a4fe867dfb-config-volume\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.523713 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a154ea0c-86e0-4615-8314-450a19397b28-config\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.523713 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3935d47-1a3e-4d52-b644-62a4fe867dfb-tmp-dir\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.523713 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.523628 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls podName:0230b3e2-67e8-4a05-bfd1-0ad3c28d3512 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:11.023612611 +0000 UTC m=+36.170428866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-gsts5" (UID: "0230b3e2-67e8-4a05-bfd1-0ad3c28d3512") : secret "samples-operator-tls" not found Aug 04 18:11:10.523911 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.523878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc3a9444-8ab5-4152-aadd-c700368fc215-config\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.524093 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.524072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a154ea0c-86e0-4615-8314-450a19397b28-trusted-ca\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.525763 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.525693 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a154ea0c-86e0-4615-8314-450a19397b28-serving-cert\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.526146 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.526112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55763a8-3b0c-4f5a-8e10-bc6964591a81-cert\") pod \"ingress-canary-94wjp\" (UID: \"b55763a8-3b0c-4f5a-8e10-bc6964591a81\") " pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.526822 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.526798 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc3a9444-8ab5-4152-aadd-c700368fc215-serving-cert\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.534968 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.534945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xxp\" (UniqueName: \"kubernetes.io/projected/b55763a8-3b0c-4f5a-8e10-bc6964591a81-kube-api-access-z6xxp\") pod \"ingress-canary-94wjp\" (UID: \"b55763a8-3b0c-4f5a-8e10-bc6964591a81\") " pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.535268 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.535246 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4944n\" (UniqueName: \"kubernetes.io/projected/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-kube-api-access-4944n\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:10.535268 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.535265 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xz7\" (UniqueName: \"kubernetes.io/projected/dc3a9444-8ab5-4152-aadd-c700368fc215-kube-api-access-p9xz7\") pod \"service-ca-operator-784f97bcf6-std9x\" (UID: \"dc3a9444-8ab5-4152-aadd-c700368fc215\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.538503 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.538484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrq5\" (UniqueName: \"kubernetes.io/projected/a154ea0c-86e0-4615-8314-450a19397b28-kube-api-access-whrq5\") pod \"console-operator-5ccd6b445f-8hh79\" (UID: \"a154ea0c-86e0-4615-8314-450a19397b28\") " pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.540001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.539984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmgm\" (UniqueName: \"kubernetes.io/projected/b3935d47-1a3e-4d52-b644-62a4fe867dfb-kube-api-access-zhmgm\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:10.540875 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.540858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" Aug 04 18:11:10.580187 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.580153 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" Aug 04 18:11:10.587351 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.587326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-m7zkx" Aug 04 18:11:10.614845 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.614299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" Aug 04 18:11:10.624102 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.623901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:10.624329 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.624302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9x4\" (UniqueName: \"kubernetes.io/projected/b6f687fc-950c-4152-8da1-ba0884fcd082-kube-api-access-ts9x4\") pod \"network-check-source-65bdbbff45-xkj75\" (UID: \"b6f687fc-950c-4152-8da1-ba0884fcd082\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" Aug 04 18:11:10.631038 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.630816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-94wjp" Aug 04 18:11:10.644257 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.644190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9x4\" (UniqueName: \"kubernetes.io/projected/b6f687fc-950c-4152-8da1-ba0884fcd082-kube-api-access-ts9x4\") pod \"network-check-source-65bdbbff45-xkj75\" (UID: \"b6f687fc-950c-4152-8da1-ba0884fcd082\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" Aug 04 18:11:10.698817 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.698470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" Aug 04 18:11:10.757412 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.757038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v"] Aug 04 18:11:10.798822 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:10.798607 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d6a004_71d5_46dc_b7bb_06bfa86f8131.slice/crio-ba683ef7a9bd98e9e60ab6e338ddd874ac85fd65b82831d3cebf04f436d0e076 WatchSource:0}: Error finding container ba683ef7a9bd98e9e60ab6e338ddd874ac85fd65b82831d3cebf04f436d0e076: Status 404 returned error can't find the container with id ba683ef7a9bd98e9e60ab6e338ddd874ac85fd65b82831d3cebf04f436d0e076 Aug 04 18:11:10.816827 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.816746 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm"] Aug 04 18:11:10.851465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.851058 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-m7zkx"] Aug 04 18:11:10.875348 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.875315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-94wjp"] Aug 04 18:11:10.886134 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.886107 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x"] Aug 04 18:11:10.890161 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:10.890095 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3a9444_8ab5_4152_aadd_c700368fc215.slice/crio-51dc97564d3444fa002daddd4b9e1d9185d04ed34c81bfdbdab50a700a2f816c WatchSource:0}: Error finding container 51dc97564d3444fa002daddd4b9e1d9185d04ed34c81bfdbdab50a700a2f816c: Status 404 returned error can't find the container with id 51dc97564d3444fa002daddd4b9e1d9185d04ed34c81bfdbdab50a700a2f816c Aug 04 18:11:10.908447 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.908348 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-8hh79"] Aug 04 18:11:10.912236 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:10.912202 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda154ea0c_86e0_4615_8314_450a19397b28.slice/crio-5bd66ddb43de52444cf61e0fd8ffdef46dceb2a9370c0807383fc7fa8dd12970 WatchSource:0}: Error finding container 5bd66ddb43de52444cf61e0fd8ffdef46dceb2a9370c0807383fc7fa8dd12970: Status 404 returned error can't find the container with id 5bd66ddb43de52444cf61e0fd8ffdef46dceb2a9370c0807383fc7fa8dd12970 Aug 04 18:11:10.928484 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.928452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.928624 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.928563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:10.928624 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.928611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:10.928714 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928637 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:11.928613927 +0000 UTC m=+37.075430165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : configmap references non-existent config key: service-ca.crt Aug 04 18:11:10.928714 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928701 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:10.928796 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928715 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fbdcb99f-qzxkm: secret "image-registry-tls" not found Aug 04 18:11:10.928796 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928715 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 18:11:10.928796 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.928717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:10.928796 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928758 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls podName:5af49ee1-5ad0-4f19-8765-0ae0c18acb08 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:11.92874356 +0000 UTC m=+37.075559799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls") pod "image-registry-76fbdcb99f-qzxkm" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08") : secret "image-registry-tls" not found Aug 04 18:11:10.928796 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928778 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:11.928769494 +0000 UTC m=+37.075585735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : secret "router-metrics-certs-default" not found Aug 04 18:11:10.928796 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928781 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:10.928992 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:10.928818 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls podName:2363937d-6717-4808-85f9-de0fe414de74 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:11.928807266 +0000 UTC m=+37.075623539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-h5r49" (UID: "2363937d-6717-4808-85f9-de0fe414de74") : secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:10.940613 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:10.940584 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75"] Aug 04 18:11:10.943976 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:10.943951 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f687fc_950c_4152_8da1_ba0884fcd082.slice/crio-566f2c09f5e4a174a6d91254021fd14336139651e79398e4705afe909b0743c6 WatchSource:0}: Error finding container 566f2c09f5e4a174a6d91254021fd14336139651e79398e4705afe909b0743c6: Status 404 returned error can't find the container with id 566f2c09f5e4a174a6d91254021fd14336139651e79398e4705afe909b0743c6 Aug 04 18:11:11.029273 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.029155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:11.029273 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.029205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:11.029496 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.029315 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:11.029496 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.029354 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 18:11:11.029496 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.029422 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls podName:b3935d47-1a3e-4d52-b644-62a4fe867dfb nodeName:}" failed. No retries permitted until 2026-08-04 18:11:12.029402021 +0000 UTC m=+37.176218262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls") pod "dns-default-cvcw2" (UID: "b3935d47-1a3e-4d52-b644-62a4fe867dfb") : secret "dns-default-metrics-tls" not found Aug 04 18:11:11.029496 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.029456 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls podName:0230b3e2-67e8-4a05-bfd1-0ad3c28d3512 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:12.029439561 +0000 UTC m=+37.176255799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-gsts5" (UID: "0230b3e2-67e8-4a05-bfd1-0ad3c28d3512") : secret "samples-operator-tls" not found Aug 04 18:11:11.343349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.343271 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:11.343507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.343272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:11.343507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.343418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:11.346828 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.346806 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wz6f8\"" Aug 04 18:11:11.347609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.346881 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 18:11:11.347609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.346886 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 18:11:11.348087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.348068 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kp5sr\"" Aug 04 18:11:11.557036 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.556702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" event={"ID":"dc3a9444-8ab5-4152-aadd-c700368fc215","Type":"ContainerStarted","Data":"51dc97564d3444fa002daddd4b9e1d9185d04ed34c81bfdbdab50a700a2f816c"} Aug 04 18:11:11.559865 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.559130 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-m7zkx" event={"ID":"b36af2a3-d7d0-469a-a78d-c20e9ffcb746","Type":"ContainerStarted","Data":"0a3b9353e193ca9088b6339ec6544d311868a00406d56790aba1c4dd0dd403de"} Aug 04 18:11:11.566670 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.566577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" event={"ID":"1fb0ddae-2f89-4c20-b7dc-12822928ddb6","Type":"ContainerStarted","Data":"9c14e81e25cc2be88cb72a4333d05e3458967c66019f386dcea53aebf8058705"} Aug 04 18:11:11.568726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.568695 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" event={"ID":"f8d6a004-71d5-46dc-b7bb-06bfa86f8131","Type":"ContainerStarted","Data":"ba683ef7a9bd98e9e60ab6e338ddd874ac85fd65b82831d3cebf04f436d0e076"} Aug 04 18:11:11.570350 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.570286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" event={"ID":"a154ea0c-86e0-4615-8314-450a19397b28","Type":"ContainerStarted","Data":"5bd66ddb43de52444cf61e0fd8ffdef46dceb2a9370c0807383fc7fa8dd12970"} Aug 04 18:11:11.571880 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.571816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" event={"ID":"b6f687fc-950c-4152-8da1-ba0884fcd082","Type":"ContainerStarted","Data":"566f2c09f5e4a174a6d91254021fd14336139651e79398e4705afe909b0743c6"} Aug 04 18:11:11.573400 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.573380 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-94wjp" event={"ID":"b55763a8-3b0c-4f5a-8e10-bc6964591a81","Type":"ContainerStarted","Data":"7f9337da8ff6ada0d84644f6f96c4e1046299714d5245d82462981c4fbcef91d"} Aug 04 18:11:11.939354 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.939306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:11.939573 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.939411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:11.939573 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.939470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:11.939573 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:11.939541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:11.939898 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.939876 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:13.939795123 +0000 UTC m=+39.086611361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : configmap references non-existent config key: service-ca.crt Aug 04 18:11:11.940472 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940448 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 18:11:11.940584 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940529 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:13.940511473 +0000 UTC m=+39.087327716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : secret "router-metrics-certs-default" not found Aug 04 18:11:11.940982 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940709 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:11.940982 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940730 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fbdcb99f-qzxkm: secret "image-registry-tls" not found Aug 04 18:11:11.940982 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940768 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls podName:5af49ee1-5ad0-4f19-8765-0ae0c18acb08 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:13.940755801 +0000 UTC m=+39.087572042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls") pod "image-registry-76fbdcb99f-qzxkm" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08") : secret "image-registry-tls" not found Aug 04 18:11:11.940982 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940848 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:11.940982 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:11.940943 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls podName:2363937d-6717-4808-85f9-de0fe414de74 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:13.940914398 +0000 UTC m=+39.087730639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-h5r49" (UID: "2363937d-6717-4808-85f9-de0fe414de74") : secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:12.041334 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:12.040508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:12.041334 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:12.040567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:12.041334 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:12.040780 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 18:11:12.041334 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:12.040844 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls podName:0230b3e2-67e8-4a05-bfd1-0ad3c28d3512 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:14.04082595 +0000 UTC m=+39.187642208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-gsts5" (UID: "0230b3e2-67e8-4a05-bfd1-0ad3c28d3512") : secret "samples-operator-tls" not found Aug 04 18:11:12.041334 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:12.041245 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:12.041334 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:12.041294 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls podName:b3935d47-1a3e-4d52-b644-62a4fe867dfb nodeName:}" failed. No retries permitted until 2026-08-04 18:11:14.041277683 +0000 UTC m=+39.188093927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls") pod "dns-default-cvcw2" (UID: "b3935d47-1a3e-4d52-b644-62a4fe867dfb") : secret "dns-default-metrics-tls" not found Aug 04 18:11:13.586264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:13.585846 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" event={"ID":"1fb0ddae-2f89-4c20-b7dc-12822928ddb6","Type":"ContainerStarted","Data":"e8313a190806dd79aeec145ba00633ad8bff38e65d4166d570542c683808e895"} Aug 04 18:11:13.605652 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:13.605505 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-s7tzm" podStartSLOduration=1.35578567 podStartE2EDuration="3.605489167s" totalImagesPullingTime="2.249703497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:13.604277341 +0000 UTC m=+38.751093628" watchObservedRunningTime="2026-08-04 18:11:13.605489167 +0000 UTC m=+38.752305426" Aug 04 18:11:13.962118 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:13.962074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:13.962287 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:13.962188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:13.962287 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962237 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:13.962287 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:13.962281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:13.962469 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962308 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls podName:2363937d-6717-4808-85f9-de0fe414de74 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:17.962287935 +0000 UTC m=+43.109104177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-h5r49" (UID: "2363937d-6717-4808-85f9-de0fe414de74") : secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:13.962469 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962376 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 18:11:13.962469 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962378 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:17.96234518 +0000 UTC m=+43.109161432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : configmap references non-existent config key: service-ca.crt Aug 04 18:11:13.962469 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962417 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:17.962402999 +0000 UTC m=+43.109219240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : secret "router-metrics-certs-default" not found Aug 04 18:11:13.962469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:13.962435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:13.962731 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962568 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:13.962731 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962583 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fbdcb99f-qzxkm: secret "image-registry-tls" not found Aug 04 18:11:13.962731 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:13.962621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls podName:5af49ee1-5ad0-4f19-8765-0ae0c18acb08 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:17.962608873 +0000 UTC m=+43.109425117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls") pod "image-registry-76fbdcb99f-qzxkm" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08") : secret "image-registry-tls" not found Aug 04 18:11:14.063336 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:14.063293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:14.063532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:14.063350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:14.063625 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:14.063603 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 18:11:14.063750 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:14.063736 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:14.063825 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:14.063800 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls podName:b3935d47-1a3e-4d52-b644-62a4fe867dfb nodeName:}" failed. No retries permitted until 2026-08-04 18:11:18.063779189 +0000 UTC m=+43.210595431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls") pod "dns-default-cvcw2" (UID: "b3935d47-1a3e-4d52-b644-62a4fe867dfb") : secret "dns-default-metrics-tls" not found Aug 04 18:11:14.064190 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:14.064165 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls podName:0230b3e2-67e8-4a05-bfd1-0ad3c28d3512 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:18.064145279 +0000 UTC m=+43.210961518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-gsts5" (UID: "0230b3e2-67e8-4a05-bfd1-0ad3c28d3512") : secret "samples-operator-tls" not found Aug 04 18:11:18.000155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:18.000102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:18.000174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:18.000219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:18.000277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000275 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000318 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000417 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fbdcb99f-qzxkm: secret "image-registry-tls" not found Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000428 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:26.000409003 +0000 UTC m=+51.147225241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : configmap references non-existent config key: service-ca.crt Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000330 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000456 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls podName:5af49ee1-5ad0-4f19-8765-0ae0c18acb08 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:26.000445095 +0000 UTC m=+51.147261338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls") pod "image-registry-76fbdcb99f-qzxkm" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08") : secret "image-registry-tls" not found Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000471 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls podName:2363937d-6717-4808-85f9-de0fe414de74 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:26.000463711 +0000 UTC m=+51.147279971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-h5r49" (UID: "2363937d-6717-4808-85f9-de0fe414de74") : secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:18.000668 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.000489 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:26.000481759 +0000 UTC m=+51.147298017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : secret "router-metrics-certs-default" not found Aug 04 18:11:18.101414 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:18.101375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:18.101600 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:18.101439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:18.101600 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.101501 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:18.101600 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.101580 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls podName:b3935d47-1a3e-4d52-b644-62a4fe867dfb nodeName:}" failed. No retries permitted until 2026-08-04 18:11:26.101559545 +0000 UTC m=+51.248375798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls") pod "dns-default-cvcw2" (UID: "b3935d47-1a3e-4d52-b644-62a4fe867dfb") : secret "dns-default-metrics-tls" not found Aug 04 18:11:18.101765 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.101600 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 18:11:18.101765 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:18.101646 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls podName:0230b3e2-67e8-4a05-bfd1-0ad3c28d3512 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:26.101633758 +0000 UTC m=+51.248449996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-gsts5" (UID: "0230b3e2-67e8-4a05-bfd1-0ad3c28d3512") : secret "samples-operator-tls" not found Aug 04 18:11:21.609349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.609315 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/0.log" Aug 04 18:11:21.609832 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.609394 2575 generic.go:356] "Generic (PLEG): container finished" podID="a154ea0c-86e0-4615-8314-450a19397b28" containerID="7109febf1810519de822767bdd63ed78cabf890d2b1c682433e240fe6b448815" exitCode=255 Aug 04 18:11:21.609832 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.609504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" event={"ID":"a154ea0c-86e0-4615-8314-450a19397b28","Type":"ContainerDied","Data":"7109febf1810519de822767bdd63ed78cabf890d2b1c682433e240fe6b448815"} Aug 04 18:11:21.609832 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.609720 2575 scope.go:117] "RemoveContainer" containerID="7109febf1810519de822767bdd63ed78cabf890d2b1c682433e240fe6b448815" Aug 04 18:11:21.613415 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.613357 2575 generic.go:356] "Generic (PLEG): container finished" podID="286e235a-d4c1-4bf7-aa18-e2b7704b6d68" containerID="15d7110f4a69c666460df7316202f1de41c76f44d4b51d39ef6af3c2e5c76a78" exitCode=0 Aug 04 18:11:21.613415 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.613397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerDied","Data":"15d7110f4a69c666460df7316202f1de41c76f44d4b51d39ef6af3c2e5c76a78"} Aug 04 18:11:21.615785 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.615409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" event={"ID":"b6f687fc-950c-4152-8da1-ba0884fcd082","Type":"ContainerStarted","Data":"d4d12da9ccf4facdb263fa18ea3128fe2b1ad67f68744f16b46143ef6d5b1a1b"} Aug 04 18:11:21.617256 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.617155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-94wjp" event={"ID":"b55763a8-3b0c-4f5a-8e10-bc6964591a81","Type":"ContainerStarted","Data":"a4cae0f2b3b574109d913de8aa8bb7d91ed2b685190118ef53859d5ea7929e0f"} Aug 04 18:11:21.618732 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.618701 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" event={"ID":"dc3a9444-8ab5-4152-aadd-c700368fc215","Type":"ContainerStarted","Data":"771a8e379ec0e5c5cbd4a18dbe8590656a082ae658e080f2005ee692b5eabc6a"} Aug 04 18:11:21.622783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.622757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-m7zkx" event={"ID":"b36af2a3-d7d0-469a-a78d-c20e9ffcb746","Type":"ContainerStarted","Data":"4454cef1aac2bfd032f4ba527496308c06ac22baef6f3f95d62f1ff51cb5a781"} Aug 04 18:11:21.624344 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.624323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" event={"ID":"f8d6a004-71d5-46dc-b7bb-06bfa86f8131","Type":"ContainerStarted","Data":"1521f24ca014c4f78c17cc5b322ac15686c29c4b142480b8dce4e3cdf14fb4a2"} Aug 04 18:11:21.647465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.647417 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" podStartSLOduration=1.8077322470000001 podStartE2EDuration="11.64740072s" totalImagesPullingTime="9.839668473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:21.64704698 +0000 UTC m=+46.793863245" watchObservedRunningTime="2026-08-04 18:11:21.64740072 +0000 UTC m=+46.794216985" Aug 04 18:11:21.702693 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.699382 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-94wjp" podStartSLOduration=1.846190904 podStartE2EDuration="11.699336237s" totalImagesPullingTime="9.853145333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:21.697942005 +0000 UTC m=+46.844758257" watchObservedRunningTime="2026-08-04 18:11:21.699336237 +0000 UTC m=+46.846152499" Aug 04 18:11:21.720547 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.718664 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-xkj75" podStartSLOduration=1.9332677820000002 podStartE2EDuration="11.718644658s" totalImagesPullingTime="9.785376876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:21.717750727 +0000 UTC m=+46.864566989" watchObservedRunningTime="2026-08-04 18:11:21.718644658 +0000 UTC m=+46.865460918" Aug 04 18:11:21.741547 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.741495 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" podStartSLOduration=1.8115912079999998 podStartE2EDuration="11.741473264s" totalImagesPullingTime="9.929882056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:21.737721195 +0000 UTC m=+46.884537455" watchObservedRunningTime="2026-08-04 18:11:21.741473264 +0000 UTC m=+46.888289525" Aug 04 18:11:21.765971 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:21.765907 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-m7zkx" podStartSLOduration=1.894336942 podStartE2EDuration="11.765879253s" totalImagesPullingTime="9.871542311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:21.76403625 +0000 UTC m=+46.910852511" watchObservedRunningTime="2026-08-04 18:11:21.765879253 +0000 UTC m=+46.912695515" Aug 04 18:11:22.162579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.162493 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k5c8f"] Aug 04 18:11:22.179290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.179255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k5c8f"] Aug 04 18:11:22.179476 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.179404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.184789 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.184762 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 18:11:22.184997 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.184981 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 18:11:22.185201 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.185185 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6h4z\"" Aug 04 18:11:22.339377 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.339310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.339556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.339391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.339556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.339450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e45e02c6-0783-4ecd-ae77-1ada65667e4c-data-volume\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.339556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.339503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e45e02c6-0783-4ecd-ae77-1ada65667e4c-crio-socket\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.339684 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.339601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79qg\" (UniqueName: \"kubernetes.io/projected/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-api-access-k79qg\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.440869 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.440832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.440869 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.440873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.440910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e45e02c6-0783-4ecd-ae77-1ada65667e4c-data-volume\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.440947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e45e02c6-0783-4ecd-ae77-1ada65667e4c-crio-socket\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.440983 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.441014 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.441060 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:22.94103691 +0000 UTC m=+48.087853150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : configmap "kube-rbac-proxy" not found Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.441080 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:22.941070567 +0000 UTC m=+48.087886805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:22.441121 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.441102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79qg\" (UniqueName: \"kubernetes.io/projected/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-api-access-k79qg\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.441477 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.441183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e45e02c6-0783-4ecd-ae77-1ada65667e4c-crio-socket\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.441477 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.441310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e45e02c6-0783-4ecd-ae77-1ada65667e4c-data-volume\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.472610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.472577 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79qg\" (UniqueName: \"kubernetes.io/projected/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-api-access-k79qg\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.630547 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.630506 2575 generic.go:356] "Generic (PLEG): container finished" podID="286e235a-d4c1-4bf7-aa18-e2b7704b6d68" containerID="04730ea50bfeb1c7f30b9d27fcf9addc1cb1f8a9386172e2c08d376d8ea872ce" exitCode=0 Aug 04 18:11:22.631064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.630627 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerDied","Data":"04730ea50bfeb1c7f30b9d27fcf9addc1cb1f8a9386172e2c08d376d8ea872ce"} Aug 04 18:11:22.633565 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.633546 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:11:22.633975 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.633959 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/0.log" Aug 04 18:11:22.634049 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.633993 2575 generic.go:356] "Generic (PLEG): container finished" podID="a154ea0c-86e0-4615-8314-450a19397b28" containerID="3da0a194855fdb6ecf7ab50c6d48a806b5a9e756fc288c12572f9580a7c2a166" exitCode=255 Aug 04 18:11:22.634049 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.634031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" event={"ID":"a154ea0c-86e0-4615-8314-450a19397b28","Type":"ContainerDied","Data":"3da0a194855fdb6ecf7ab50c6d48a806b5a9e756fc288c12572f9580a7c2a166"} Aug 04 18:11:22.634145 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.634070 2575 scope.go:117] "RemoveContainer" containerID="7109febf1810519de822767bdd63ed78cabf890d2b1c682433e240fe6b448815" Aug 04 18:11:22.634317 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.634302 2575 scope.go:117] "RemoveContainer" containerID="3da0a194855fdb6ecf7ab50c6d48a806b5a9e756fc288c12572f9580a7c2a166" Aug 04 18:11:22.634519 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.634491 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-8hh79_openshift-console-operator(a154ea0c-86e0-4615-8314-450a19397b28)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" podUID="a154ea0c-86e0-4615-8314-450a19397b28" Aug 04 18:11:22.946314 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.946279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.946472 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:22.946323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:22.946472 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.946439 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 18:11:22.946472 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.946461 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:22.946574 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.946540 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:23.946506313 +0000 UTC m=+49.093322552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : configmap "kube-rbac-proxy" not found Aug 04 18:11:22.946574 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:22.946556 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:23.946549423 +0000 UTC m=+49.093365661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:23.639108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:23.639079 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:11:23.639582 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:23.639458 2575 scope.go:117] "RemoveContainer" containerID="3da0a194855fdb6ecf7ab50c6d48a806b5a9e756fc288c12572f9580a7c2a166" Aug 04 18:11:23.639685 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:23.639665 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-8hh79_openshift-console-operator(a154ea0c-86e0-4615-8314-450a19397b28)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" podUID="a154ea0c-86e0-4615-8314-450a19397b28" Aug 04 18:11:23.642329 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:23.642303 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c896x" event={"ID":"286e235a-d4c1-4bf7-aa18-e2b7704b6d68","Type":"ContainerStarted","Data":"a785f763e72cbd2d513f3e64d1ca8e7f4620444b56f0293cc6507af9d0d43081"} Aug 04 18:11:23.680537 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:23.680494 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c896x" podStartSLOduration=7.372439782 podStartE2EDuration="47.680481203s" totalImagesPullingTime="40.308041421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:23.679589869 +0000 UTC m=+48.826406129" watchObservedRunningTime="2026-08-04 18:11:23.680481203 +0000 UTC m=+48.827297462" Aug 04 18:11:23.958494 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:23.958459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:23.958494 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:23.958498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:23.958697 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:23.958572 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 18:11:23.958697 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:23.958608 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:23.958697 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:23.958620 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:25.958606481 +0000 UTC m=+51.105422719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : configmap "kube-rbac-proxy" not found Aug 04 18:11:23.958799 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:23.958775 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:25.958753517 +0000 UTC m=+51.105569760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:24.294612 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:24.294534 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjjg9_5f8c2fcb-e227-4861-8e7a-5152a4ed3d24/dns-node-resolver/0.log" Aug 04 18:11:25.293402 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:25.293350 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xn6wp_b4763dc2-4f1b-4be6-a14b-0bdb04fd9206/node-ca/0.log" Aug 04 18:11:25.695219 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:25.695190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-94wjp_b55763a8-3b0c-4f5a-8e10-bc6964591a81/serve-healthcheck-canary/0.log" Aug 04 18:11:25.974862 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:25.974770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:25.974862 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:25.974810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:25.975055 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:25.974914 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:25.975055 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:25.974980 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:29.974962531 +0000 UTC m=+55.121778769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:25.975420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:25.975401 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e45e02c6-0783-4ecd-ae77-1ada65667e4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:26.075222 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:26.075192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:26.075424 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:26.075242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:26.075424 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075332 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:26.075424 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075348 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 18:11:26.075424 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075391 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76fbdcb99f-qzxkm: secret "image-registry-tls" not found Aug 04 18:11:26.075424 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075406 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls podName:2363937d-6717-4808-85f9-de0fe414de74 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:42.075393225 +0000 UTC m=+67.222209462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-h5r49" (UID: "2363937d-6717-4808-85f9-de0fe414de74") : secret "cluster-monitoring-operator-tls" not found Aug 04 18:11:26.075639 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:26.075428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:26.075639 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075440 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls podName:5af49ee1-5ad0-4f19-8765-0ae0c18acb08 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:42.075424574 +0000 UTC m=+67.222240817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls") pod "image-registry-76fbdcb99f-qzxkm" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08") : secret "image-registry-tls" not found Aug 04 18:11:26.075639 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075476 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:42.075469634 +0000 UTC m=+67.222285872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : configmap references non-existent config key: service-ca.crt Aug 04 18:11:26.075639 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:26.075515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:26.075639 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075579 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 18:11:26.075639 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.075600 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs podName:40806ebc-9194-4588-93ba-7f9563faaf25 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:42.075593225 +0000 UTC m=+67.222409462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs") pod "router-default-69fbf496bb-v6nzt" (UID: "40806ebc-9194-4588-93ba-7f9563faaf25") : secret "router-metrics-certs-default" not found Aug 04 18:11:26.176208 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:26.176177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:26.176358 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:26.176220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:26.176358 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.176339 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 18:11:26.176358 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.176342 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 18:11:26.176510 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.176410 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls podName:0230b3e2-67e8-4a05-bfd1-0ad3c28d3512 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:42.176395042 +0000 UTC m=+67.323211282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-gsts5" (UID: "0230b3e2-67e8-4a05-bfd1-0ad3c28d3512") : secret "samples-operator-tls" not found Aug 04 18:11:26.176510 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:26.176430 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls podName:b3935d47-1a3e-4d52-b644-62a4fe867dfb nodeName:}" failed. No retries permitted until 2026-08-04 18:11:42.176416314 +0000 UTC m=+67.323232552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls") pod "dns-default-cvcw2" (UID: "b3935d47-1a3e-4d52-b644-62a4fe867dfb") : secret "dns-default-metrics-tls" not found Aug 04 18:11:30.009952 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:30.009901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:30.010572 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:30.010055 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 18:11:30.010572 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:30.010130 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls podName:e45e02c6-0783-4ecd-ae77-1ada65667e4c nodeName:}" failed. No retries permitted until 2026-08-04 18:11:38.010113439 +0000 UTC m=+63.156929677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5c8f" (UID: "e45e02c6-0783-4ecd-ae77-1ada65667e4c") : secret "insights-runtime-extractor-tls" not found Aug 04 18:11:30.624776 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:30.624739 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:30.624976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:30.624810 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:30.625245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:30.625226 2575 scope.go:117] "RemoveContainer" containerID="3da0a194855fdb6ecf7ab50c6d48a806b5a9e756fc288c12572f9580a7c2a166" Aug 04 18:11:30.625460 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:30.625436 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-8hh79_openshift-console-operator(a154ea0c-86e0-4615-8314-450a19397b28)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" podUID="a154ea0c-86e0-4615-8314-450a19397b28" Aug 04 18:11:37.552812 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:37.552784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pcjf4" Aug 04 18:11:38.078961 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.078916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:38.081558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.081529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e45e02c6-0783-4ecd-ae77-1ada65667e4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5c8f\" (UID: \"e45e02c6-0783-4ecd-ae77-1ada65667e4c\") " pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:38.094178 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.094157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r6h4z\"" Aug 04 18:11:38.102459 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.102435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k5c8f" Aug 04 18:11:38.236754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.236725 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k5c8f"] Aug 04 18:11:38.239150 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:38.239120 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45e02c6_0783_4ecd_ae77_1ada65667e4c.slice/crio-2583e522d9e25468dc8891fe94908a7919e6670db3703b40e468068c220363da WatchSource:0}: Error finding container 2583e522d9e25468dc8891fe94908a7919e6670db3703b40e468068c220363da: Status 404 returned error can't find the container with id 2583e522d9e25468dc8891fe94908a7919e6670db3703b40e468068c220363da Aug 04 18:11:38.696249 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.696213 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5c8f" event={"ID":"e45e02c6-0783-4ecd-ae77-1ada65667e4c","Type":"ContainerStarted","Data":"218fb4fe93e137ecee3478dee2b44a24ad2db02e9077e6bfbc97e947a15ec873"} Aug 04 18:11:38.696631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:38.696269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5c8f" event={"ID":"e45e02c6-0783-4ecd-ae77-1ada65667e4c","Type":"ContainerStarted","Data":"2583e522d9e25468dc8891fe94908a7919e6670db3703b40e468068c220363da"} Aug 04 18:11:39.701187 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:39.701148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5c8f" event={"ID":"e45e02c6-0783-4ecd-ae77-1ada65667e4c","Type":"ContainerStarted","Data":"810d444bc0eedd3b07a23529b8314da879a05b986e5e13b412689675bde6950b"} Aug 04 18:11:40.098749 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.098678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:40.101763 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.101746 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 18:11:40.112101 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.112080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a17151a-4a1c-4988-95f7-8a3b293cf489-metrics-certs\") pod \"network-metrics-daemon-zv8cr\" (UID: \"9a17151a-4a1c-4988-95f7-8a3b293cf489\") " pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:40.176075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.176050 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wz6f8\"" Aug 04 18:11:40.184380 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.184343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zv8cr" Aug 04 18:11:40.199700 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.199663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:40.202548 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.202530 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 18:11:40.213625 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.213602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2d36207-38b9-4ace-a64d-f0439dea7bdc-original-pull-secret\") pod \"global-pull-secret-syncer-4fkfc\" (UID: \"c2d36207-38b9-4ace-a64d-f0439dea7bdc\") " pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:40.301097 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.301060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:40.303747 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.303722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp74s\" (UniqueName: \"kubernetes.io/projected/ff35f4cf-3ecb-4ac5-8132-28e836321b8d-kube-api-access-cp74s\") pod \"network-check-target-fwqwx\" (UID: \"ff35f4cf-3ecb-4ac5-8132-28e836321b8d\") " pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:40.318192 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.318168 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zv8cr"] Aug 04 18:11:40.322487 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:40.322460 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a17151a_4a1c_4988_95f7_8a3b293cf489.slice/crio-909cb5d07417e40442651bb019698382b4df9b3dedaef1390c3c9530dbebd8a6 WatchSource:0}: Error finding container 909cb5d07417e40442651bb019698382b4df9b3dedaef1390c3c9530dbebd8a6: Status 404 returned error can't find the container with id 909cb5d07417e40442651bb019698382b4df9b3dedaef1390c3c9530dbebd8a6 Aug 04 18:11:40.457668 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.457633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fkfc" Aug 04 18:11:40.469730 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.469707 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kp5sr\"" Aug 04 18:11:40.477925 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.477900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:40.613011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.612980 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fkfc"] Aug 04 18:11:40.629526 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.629497 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fwqwx"] Aug 04 18:11:40.706232 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:40.706202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zv8cr" event={"ID":"9a17151a-4a1c-4988-95f7-8a3b293cf489","Type":"ContainerStarted","Data":"909cb5d07417e40442651bb019698382b4df9b3dedaef1390c3c9530dbebd8a6"} Aug 04 18:11:40.896080 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:40.896009 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d36207_38b9_4ace_a64d_f0439dea7bdc.slice/crio-53ec2b18ca15b7a1601af0423c56d9ea7bbf23ed5b9e88e7d25dc39e03499007 WatchSource:0}: Error finding container 53ec2b18ca15b7a1601af0423c56d9ea7bbf23ed5b9e88e7d25dc39e03499007: Status 404 returned error can't find the container with id 53ec2b18ca15b7a1601af0423c56d9ea7bbf23ed5b9e88e7d25dc39e03499007 Aug 04 18:11:40.896751 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:40.896722 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff35f4cf_3ecb_4ac5_8132_28e836321b8d.slice/crio-70b18af9bfed3838d69d0c8897f857ec18cf2e2856a65d2631c31799246a2829 WatchSource:0}: Error finding container 70b18af9bfed3838d69d0c8897f857ec18cf2e2856a65d2631c31799246a2829: Status 404 returned error can't find the container with id 70b18af9bfed3838d69d0c8897f857ec18cf2e2856a65d2631c31799246a2829 Aug 04 18:11:41.713939 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.713893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5c8f" event={"ID":"e45e02c6-0783-4ecd-ae77-1ada65667e4c","Type":"ContainerStarted","Data":"96c61b631dd7f2e97645f3113fd48a1b4129f3db8b40c52909aedc220c70d5e6"} Aug 04 18:11:41.715561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.715534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fkfc" event={"ID":"c2d36207-38b9-4ace-a64d-f0439dea7bdc","Type":"ContainerStarted","Data":"53ec2b18ca15b7a1601af0423c56d9ea7bbf23ed5b9e88e7d25dc39e03499007"} Aug 04 18:11:41.717330 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.717300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fwqwx" event={"ID":"ff35f4cf-3ecb-4ac5-8132-28e836321b8d","Type":"ContainerStarted","Data":"6228ec106d9ab7604ae50e1c1ad7451f3e60e6f7026e274340755ac8570a38fd"} Aug 04 18:11:41.717456 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.717347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fwqwx" event={"ID":"ff35f4cf-3ecb-4ac5-8132-28e836321b8d","Type":"ContainerStarted","Data":"70b18af9bfed3838d69d0c8897f857ec18cf2e2856a65d2631c31799246a2829"} Aug 04 18:11:41.717456 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.717431 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:11:41.739003 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.738956 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k5c8f" podStartSLOduration=17.155697183 podStartE2EDuration="19.738937528s" totalImagesPullingTime="2.583240345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:11:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:41.737623797 +0000 UTC m=+66.884440058" watchObservedRunningTime="2026-08-04 18:11:41.738937528 +0000 UTC m=+66.885753790" Aug 04 18:11:41.768087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:41.768040 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fwqwx" podStartSLOduration=65.768026515 podStartE2EDuration="1m5.768026515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:41.768006201 +0000 UTC m=+66.914822461" watchObservedRunningTime="2026-08-04 18:11:41.768026515 +0000 UTC m=+66.914842776" Aug 04 18:11:42.117212 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.117131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:42.117414 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.117216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:42.117414 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.117260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:42.117414 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.117301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:42.118281 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.117922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40806ebc-9194-4588-93ba-7f9563faaf25-service-ca-bundle\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:42.120563 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.120512 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2363937d-6717-4808-85f9-de0fe414de74-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-h5r49\" (UID: \"2363937d-6717-4808-85f9-de0fe414de74\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:42.120682 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.120662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"image-registry-76fbdcb99f-qzxkm\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:42.120746 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.120679 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40806ebc-9194-4588-93ba-7f9563faaf25-metrics-certs\") pod \"router-default-69fbf496bb-v6nzt\" (UID: \"40806ebc-9194-4588-93ba-7f9563faaf25\") " pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:42.218589 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.218561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:42.218728 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.218606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:42.221345 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.221320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3935d47-1a3e-4d52-b644-62a4fe867dfb-metrics-tls\") pod \"dns-default-cvcw2\" (UID: \"b3935d47-1a3e-4d52-b644-62a4fe867dfb\") " pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:42.221731 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.221702 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0230b3e2-67e8-4a05-bfd1-0ad3c28d3512-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-gsts5\" (UID: \"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:42.357775 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.357754 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg4n\"" Aug 04 18:11:42.365706 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.365683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:42.366492 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.365710 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-b6rd5\"" Aug 04 18:11:42.373892 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.373758 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:42.375428 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.375403 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-b7zzs\"" Aug 04 18:11:42.382913 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.382889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" Aug 04 18:11:42.408861 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.408835 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zlqfr\"" Aug 04 18:11:42.416056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.416025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" Aug 04 18:11:42.447961 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.443037 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-khq2z\"" Aug 04 18:11:42.447961 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.447722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:42.558814 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.558604 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76fbdcb99f-qzxkm"] Aug 04 18:11:42.564337 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:42.563533 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af49ee1_5ad0_4f19_8765_0ae0c18acb08.slice/crio-983a7d1bd604fe605813e28693e03e02f103416f39852cf877e64ea6d0188896 WatchSource:0}: Error finding container 983a7d1bd604fe605813e28693e03e02f103416f39852cf877e64ea6d0188896: Status 404 returned error can't find the container with id 983a7d1bd604fe605813e28693e03e02f103416f39852cf877e64ea6d0188896 Aug 04 18:11:42.608463 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.607140 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69fbf496bb-v6nzt"] Aug 04 18:11:42.612213 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:42.612186 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40806ebc_9194_4588_93ba_7f9563faaf25.slice/crio-b7b242c08fa20e5e87c1105f92f244aec6e726e613d6b8bc8d18be384652fe2d WatchSource:0}: Error finding container b7b242c08fa20e5e87c1105f92f244aec6e726e613d6b8bc8d18be384652fe2d: Status 404 returned error can't find the container with id b7b242c08fa20e5e87c1105f92f244aec6e726e613d6b8bc8d18be384652fe2d Aug 04 18:11:42.619796 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.619739 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49"] Aug 04 18:11:42.631028 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:42.630992 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2363937d_6717_4808_85f9_de0fe414de74.slice/crio-34e6c75e735d271e1cd66ae8213fa64be779e4d6d8c8c8d376f9917e99381194 WatchSource:0}: Error finding container 34e6c75e735d271e1cd66ae8213fa64be779e4d6d8c8c8d376f9917e99381194: Status 404 returned error can't find the container with id 34e6c75e735d271e1cd66ae8213fa64be779e4d6d8c8c8d376f9917e99381194 Aug 04 18:11:42.660107 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.659292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5"] Aug 04 18:11:42.702586 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.702513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cvcw2"] Aug 04 18:11:42.706342 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:42.706266 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3935d47_1a3e_4d52_b644_62a4fe867dfb.slice/crio-332dc401ec1b3b3a11d5fbb27e8b19d9ba887eeaa1c5ce07bd927b3b78f0cff0 WatchSource:0}: Error finding container 332dc401ec1b3b3a11d5fbb27e8b19d9ba887eeaa1c5ce07bd927b3b78f0cff0: Status 404 returned error can't find the container with id 332dc401ec1b3b3a11d5fbb27e8b19d9ba887eeaa1c5ce07bd927b3b78f0cff0 Aug 04 18:11:42.724543 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.724475 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cvcw2" event={"ID":"b3935d47-1a3e-4d52-b644-62a4fe867dfb","Type":"ContainerStarted","Data":"332dc401ec1b3b3a11d5fbb27e8b19d9ba887eeaa1c5ce07bd927b3b78f0cff0"} Aug 04 18:11:42.725873 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.725848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" event={"ID":"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512","Type":"ContainerStarted","Data":"67c34f88c24fe043c7881ca30ba69035c9757445367014450825abd13a05cc02"} Aug 04 18:11:42.728628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.728392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" event={"ID":"5af49ee1-5ad0-4f19-8765-0ae0c18acb08","Type":"ContainerStarted","Data":"05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423"} Aug 04 18:11:42.728628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.728439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" event={"ID":"5af49ee1-5ad0-4f19-8765-0ae0c18acb08","Type":"ContainerStarted","Data":"983a7d1bd604fe605813e28693e03e02f103416f39852cf877e64ea6d0188896"} Aug 04 18:11:42.728628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.728483 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:11:42.730914 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.730887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zv8cr" event={"ID":"9a17151a-4a1c-4988-95f7-8a3b293cf489","Type":"ContainerStarted","Data":"25868fb89852e1995849d34bd874b02bf8e502c9e65162cc585981ab7b466a45"} Aug 04 18:11:42.730998 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.730942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zv8cr" event={"ID":"9a17151a-4a1c-4988-95f7-8a3b293cf489","Type":"ContainerStarted","Data":"fa38ccc38ab307744472131bdae343a6704de4bbf1736cd6f31edcbb31dfb382"} Aug 04 18:11:42.732662 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.732612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" event={"ID":"2363937d-6717-4808-85f9-de0fe414de74","Type":"ContainerStarted","Data":"34e6c75e735d271e1cd66ae8213fa64be779e4d6d8c8c8d376f9917e99381194"} Aug 04 18:11:42.734977 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.734952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" event={"ID":"40806ebc-9194-4588-93ba-7f9563faaf25","Type":"ContainerStarted","Data":"5693890739ca558e13d88f931f4f4e24c464ce16446e658088f24bb7148b0516"} Aug 04 18:11:42.735059 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.735007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" event={"ID":"40806ebc-9194-4588-93ba-7f9563faaf25","Type":"ContainerStarted","Data":"b7b242c08fa20e5e87c1105f92f244aec6e726e613d6b8bc8d18be384652fe2d"} Aug 04 18:11:42.764500 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.764453 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" podStartSLOduration=66.764439012 podStartE2EDuration="1m6.764439012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:42.762992365 +0000 UTC m=+67.909808623" watchObservedRunningTime="2026-08-04 18:11:42.764439012 +0000 UTC m=+67.911255272" Aug 04 18:11:42.785795 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.785736 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" podStartSLOduration=32.785718292 podStartE2EDuration="32.785718292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:42.785030166 +0000 UTC m=+67.931846427" watchObservedRunningTime="2026-08-04 18:11:42.785718292 +0000 UTC m=+67.932534553" Aug 04 18:11:42.808003 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:42.807958 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zv8cr" podStartSLOduration=65.962828302 podStartE2EDuration="1m7.807941723s" totalImagesPullingTime="1.845113421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:42.806968977 +0000 UTC m=+67.953785236" watchObservedRunningTime="2026-08-04 18:11:42.807941723 +0000 UTC m=+67.954757985" Aug 04 18:11:43.344393 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.342856 2575 scope.go:117] "RemoveContainer" containerID="3da0a194855fdb6ecf7ab50c6d48a806b5a9e756fc288c12572f9580a7c2a166" Aug 04 18:11:43.374049 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.374018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:43.376879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.376858 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:43.745032 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.745004 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:11:43.745522 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.745200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" event={"ID":"a154ea0c-86e0-4615-8314-450a19397b28","Type":"ContainerStarted","Data":"0b73556c0619c557409d024bcab0bee3cdd3abf0d257584e5a2362244b0fd194"} Aug 04 18:11:43.746270 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.746227 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:43.746383 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.746272 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:43.747976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.747953 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69fbf496bb-v6nzt" Aug 04 18:11:43.769289 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:43.767794 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" podStartSLOduration=23.950448179 podStartE2EDuration="33.767774526s" totalImagesPullingTime="9.817326347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:43.767308643 +0000 UTC m=+68.914124912" watchObservedRunningTime="2026-08-04 18:11:43.767774526 +0000 UTC m=+68.914590790" Aug 04 18:11:44.566947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:44.566409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-8hh79" Aug 04 18:11:47.767673 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.767620 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cvcw2" event={"ID":"b3935d47-1a3e-4d52-b644-62a4fe867dfb","Type":"ContainerStarted","Data":"e4b9d6387c7f864055e867b7f298fe5c8b89e21712260aa98f0443e049100d97"} Aug 04 18:11:47.768180 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.767729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cvcw2" event={"ID":"b3935d47-1a3e-4d52-b644-62a4fe867dfb","Type":"ContainerStarted","Data":"dc81cfd6b5ecb5f3ad923402d3f4f04fce2b03ac16eaa6e969c04554bcbfacd5"} Aug 04 18:11:47.768180 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.767878 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:47.769626 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.769601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" event={"ID":"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512","Type":"ContainerStarted","Data":"99a6aca6791aa73e92cb06974e5dfe2f0cd586b7920a7b7cbbb77f7f73300fbc"} Aug 04 18:11:47.769744 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.769644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" event={"ID":"0230b3e2-67e8-4a05-bfd1-0ad3c28d3512","Type":"ContainerStarted","Data":"8d813ca3254bbc643bd92147a49139967652f2d09ad10335623708ca3a8488c7"} Aug 04 18:11:47.771036 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.771014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" event={"ID":"2363937d-6717-4808-85f9-de0fe414de74","Type":"ContainerStarted","Data":"d116c7abef0e23d5425838e48f025a36d8459540f6679dc084b334d98aec9967"} Aug 04 18:11:47.772512 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.772491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fkfc" event={"ID":"c2d36207-38b9-4ace-a64d-f0439dea7bdc","Type":"ContainerStarted","Data":"6a35201743fccc8679f97bed7fe4cc8b259612d97cab7d7ba111ac75ec10bd1c"} Aug 04 18:11:47.789167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.789123 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cvcw2" podStartSLOduration=33.549336478 podStartE2EDuration="37.789111104s" totalImagesPullingTime="4.239774626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:47.788733987 +0000 UTC m=+72.935550249" watchObservedRunningTime="2026-08-04 18:11:47.789111104 +0000 UTC m=+72.935927364" Aug 04 18:11:47.821815 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.821775 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4fkfc" podStartSLOduration=65.750261938 podStartE2EDuration="1m11.821763114s" totalImagesPullingTime="6.071501176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:47.820907489 +0000 UTC m=+72.967723748" watchObservedRunningTime="2026-08-04 18:11:47.821763114 +0000 UTC m=+72.968579373" Aug 04 18:11:47.866009 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.865949 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-h5r49" podStartSLOduration=33.550366578 podStartE2EDuration="37.865928591s" totalImagesPullingTime="4.315562013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:47.864906453 +0000 UTC m=+73.011722714" watchObservedRunningTime="2026-08-04 18:11:47.865928591 +0000 UTC m=+73.012744853" Aug 04 18:11:47.899228 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:47.899183 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-gsts5" podStartSLOduration=33.653924486 podStartE2EDuration="37.899166585s" totalImagesPullingTime="4.245242099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:47.898999361 +0000 UTC m=+73.045815621" watchObservedRunningTime="2026-08-04 18:11:47.899166585 +0000 UTC m=+73.045982844" Aug 04 18:11:54.654792 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.654757 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d5574897-qvl5c"] Aug 04 18:11:54.681435 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.681401 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d5574897-qvl5c"] Aug 04 18:11:54.681577 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.681517 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.684344 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.684319 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 18:11:54.684507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.684425 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hjg9c\"" Aug 04 18:11:54.685461 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.685441 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 18:11:54.685461 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.685456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 18:11:54.685631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.685468 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 18:11:54.685631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.685484 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 18:11:54.685631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.685488 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 18:11:54.685631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.685449 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 18:11:54.820610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.820582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-oauth-serving-cert\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.820782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.820622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-config\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.820782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.820653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dh5d\" (UniqueName: \"kubernetes.io/projected/6df7a80a-6121-46d0-9e3c-f07e37eb6137-kube-api-access-5dh5d\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.820782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.820717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-oauth-config\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.820782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.820775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-service-ca\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.820949 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.820797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-serving-cert\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.921578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.921533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-oauth-config\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.921780 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.921612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-service-ca\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.921780 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.921636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-serving-cert\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.921780 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.921668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-oauth-serving-cert\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.921780 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.921694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-config\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.921987 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.921779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dh5d\" (UniqueName: \"kubernetes.io/projected/6df7a80a-6121-46d0-9e3c-f07e37eb6137-kube-api-access-5dh5d\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.922522 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.922427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-oauth-serving-cert\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.922522 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.922467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-service-ca\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.922781 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.922759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-config\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.924214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.924192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-oauth-config\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.924296 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.924274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-serving-cert\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.931765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.931737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dh5d\" (UniqueName: \"kubernetes.io/projected/6df7a80a-6121-46d0-9e3c-f07e37eb6137-kube-api-access-5dh5d\") pod \"console-84d5574897-qvl5c\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:54.990783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:54.990744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:11:55.145427 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:55.145244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d5574897-qvl5c"] Aug 04 18:11:55.148028 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:55.148000 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df7a80a_6121_46d0_9e3c_f07e37eb6137.slice/crio-9cde794a974191ceae42703420fdc3ad4f7efdd5f8d394c73d5151f6da28239f WatchSource:0}: Error finding container 9cde794a974191ceae42703420fdc3ad4f7efdd5f8d394c73d5151f6da28239f: Status 404 returned error can't find the container with id 9cde794a974191ceae42703420fdc3ad4f7efdd5f8d394c73d5151f6da28239f Aug 04 18:11:55.802916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:55.802802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d5574897-qvl5c" event={"ID":"6df7a80a-6121-46d0-9e3c-f07e37eb6137","Type":"ContainerStarted","Data":"9cde794a974191ceae42703420fdc3ad4f7efdd5f8d394c73d5151f6da28239f"} Aug 04 18:11:57.319915 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.319872 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf"] Aug 04 18:11:57.322855 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.322827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.329499 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.329470 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 18:11:57.329697 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.329475 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 18:11:57.330058 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.330036 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v9mrj\"" Aug 04 18:11:57.330553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.330532 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 18:11:57.348831 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.348803 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf"] Aug 04 18:11:57.371694 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.371654 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-866wn"] Aug 04 18:11:57.373927 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.373901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.377569 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.377542 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 04 18:11:57.378535 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.378508 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 04 18:11:57.378686 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.378580 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 04 18:11:57.378803 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.378786 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rxrxw\"" Aug 04 18:11:57.398581 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.398555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-866wn"] Aug 04 18:11:57.426408 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.426374 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rf6pt"] Aug 04 18:11:57.429019 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.428997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.436609 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:57.436577 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-wk8r4\" is forbidden: User \"system:node:ip-10-0-142-222.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-222.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wk8r4\"" type="*v1.Secret" Aug 04 18:11:57.436794 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.436736 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 18:11:57.437503 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.437477 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 18:11:57.438529 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.438494 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 18:11:57.442873 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.442854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.442989 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.442884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.442989 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.442910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.442989 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.442939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.442989 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.442984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.443205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.443067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2dpm\" (UniqueName: \"kubernetes.io/projected/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-kube-api-access-w2dpm\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.443205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.443114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.443205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.443155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.443205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.443187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28vjg\" (UniqueName: \"kubernetes.io/projected/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-api-access-28vjg\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.443414 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.443219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.544554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-sys\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-textfile\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:57.544699 2575 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-wtmp\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:57.544742 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 04 18:11:57.544754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7c9v\" (UniqueName: \"kubernetes.io/projected/a8048dc7-8311-402c-a23e-1fcb4f629df8-kube-api-access-f7c9v\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:57.544775 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-tls podName:b7384e93-e3d1-4ab3-bf64-1e932beac4f9 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:58.044751193 +0000 UTC m=+83.191567437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-866wn" (UID: "b7384e93-e3d1-4ab3-bf64-1e932beac4f9") : secret "kube-state-metrics-tls" not found Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:11:57.544797 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-tls podName:2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:58.044789508 +0000 UTC m=+83.191605746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-qhqvf" (UID: "2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5") : secret "openshift-state-metrics-tls" not found Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-root\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2dpm\" (UniqueName: \"kubernetes.io/projected/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-kube-api-access-w2dpm\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.544983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-accelerators-collector-config\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-tls\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.545233 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28vjg\" (UniqueName: \"kubernetes.io/projected/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-api-access-28vjg\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.545628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8048dc7-8311-402c-a23e-1fcb4f629df8-metrics-client-ca\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.545628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.545628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.546766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.545992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.546766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.546285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.546766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.546705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.548194 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.548061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.550115 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.550046 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.555923 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.555894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28vjg\" (UniqueName: \"kubernetes.io/projected/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-api-access-28vjg\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:57.556127 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.556108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2dpm\" (UniqueName: \"kubernetes.io/projected/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-kube-api-access-w2dpm\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:57.646177 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-tls\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646177 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8048dc7-8311-402c-a23e-1fcb4f629df8-metrics-client-ca\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646447 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-sys\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646447 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-textfile\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646447 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-wtmp\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646447 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7c9v\" (UniqueName: \"kubernetes.io/projected/a8048dc7-8311-402c-a23e-1fcb4f629df8-kube-api-access-f7c9v\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646447 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-root\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646695 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646695 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-wtmp\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646695 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-accelerators-collector-config\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646695 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-sys\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646695 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646661 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-textfile\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.646922 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.646721 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8048dc7-8311-402c-a23e-1fcb4f629df8-root\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.647047 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.647013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-accelerators-collector-config\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.647124 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.647108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8048dc7-8311-402c-a23e-1fcb4f629df8-metrics-client-ca\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.648865 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.648838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-tls\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.649195 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.649173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8048dc7-8311-402c-a23e-1fcb4f629df8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.656351 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.656327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7c9v\" (UniqueName: \"kubernetes.io/projected/a8048dc7-8311-402c-a23e-1fcb4f629df8-kube-api-access-f7c9v\") pod \"node-exporter-rf6pt\" (UID: \"a8048dc7-8311-402c-a23e-1fcb4f629df8\") " pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:57.780557 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:57.780521 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cvcw2" Aug 04 18:11:58.050753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.050710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:58.050753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.050754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:58.053551 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.053526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7384e93-e3d1-4ab3-bf64-1e932beac4f9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-866wn\" (UID: \"b7384e93-e3d1-4ab3-bf64-1e932beac4f9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:58.053665 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.053573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-qhqvf\" (UID: \"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:58.235376 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.235324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" Aug 04 18:11:58.273465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.273438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wk8r4\"" Aug 04 18:11:58.281451 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.281422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rf6pt" Aug 04 18:11:58.283475 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.283445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" Aug 04 18:11:58.471804 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:58.471766 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8048dc7_8311_402c_a23e_1fcb4f629df8.slice/crio-c5c412c3bc703577889a590341888e63f8124e354f1b305965f090af0e8b22f5 WatchSource:0}: Error finding container c5c412c3bc703577889a590341888e63f8124e354f1b305965f090af0e8b22f5: Status 404 returned error can't find the container with id c5c412c3bc703577889a590341888e63f8124e354f1b305965f090af0e8b22f5 Aug 04 18:11:58.653777 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.653748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf"] Aug 04 18:11:58.656237 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:58.656210 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4b2eaf_5a17_456a_a8e6_8cec67a6e1b5.slice/crio-324654f0a6e7981a0f0c074c0f1b2f21785d2b67015ecc73f49bbab255b8401d WatchSource:0}: Error finding container 324654f0a6e7981a0f0c074c0f1b2f21785d2b67015ecc73f49bbab255b8401d: Status 404 returned error can't find the container with id 324654f0a6e7981a0f0c074c0f1b2f21785d2b67015ecc73f49bbab255b8401d Aug 04 18:11:58.664838 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.664813 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-866wn"] Aug 04 18:11:58.669397 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:11:58.669341 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7384e93_e3d1_4ab3_bf64_1e932beac4f9.slice/crio-2f033e2c0cc0b1a435d8c536fb6e5f9c6ef230400c56ec509135ff45f0f63c26 WatchSource:0}: Error finding container 2f033e2c0cc0b1a435d8c536fb6e5f9c6ef230400c56ec509135ff45f0f63c26: Status 404 returned error can't find the container with id 2f033e2c0cc0b1a435d8c536fb6e5f9c6ef230400c56ec509135ff45f0f63c26 Aug 04 18:11:58.818953 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.818913 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d5574897-qvl5c" event={"ID":"6df7a80a-6121-46d0-9e3c-f07e37eb6137","Type":"ContainerStarted","Data":"3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a"} Aug 04 18:11:58.820941 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.820812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" event={"ID":"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5","Type":"ContainerStarted","Data":"a23a409107fb0f2000867ff96d51e97d5eedd887a338adb79141eb9c88047131"} Aug 04 18:11:58.820941 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.820869 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" event={"ID":"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5","Type":"ContainerStarted","Data":"b61c1c27d39af18884a1d0afc7c71b3b7ae7df99ab8d437b2c7f61f155e60590"} Aug 04 18:11:58.820941 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.820901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" event={"ID":"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5","Type":"ContainerStarted","Data":"324654f0a6e7981a0f0c074c0f1b2f21785d2b67015ecc73f49bbab255b8401d"} Aug 04 18:11:58.822105 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.822084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf6pt" event={"ID":"a8048dc7-8311-402c-a23e-1fcb4f629df8","Type":"ContainerStarted","Data":"c5c412c3bc703577889a590341888e63f8124e354f1b305965f090af0e8b22f5"} Aug 04 18:11:58.823184 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.823165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" event={"ID":"b7384e93-e3d1-4ab3-bf64-1e932beac4f9","Type":"ContainerStarted","Data":"2f033e2c0cc0b1a435d8c536fb6e5f9c6ef230400c56ec509135ff45f0f63c26"} Aug 04 18:11:58.838503 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:11:58.838445 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84d5574897-qvl5c" podStartSLOduration=1.503848654 podStartE2EDuration="4.838426959s" totalImagesPullingTime="3.334578305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:58.838157268 +0000 UTC m=+83.984973540" watchObservedRunningTime="2026-08-04 18:11:58.838426959 +0000 UTC m=+83.985243220" Aug 04 18:12:00.833998 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.833903 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8048dc7-8311-402c-a23e-1fcb4f629df8" containerID="cc2706ef3964207f3b95d79eda84718d52480a4fed57c2232ac69803105bd452" exitCode=0 Aug 04 18:12:00.834456 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.833999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf6pt" event={"ID":"a8048dc7-8311-402c-a23e-1fcb4f629df8","Type":"ContainerDied","Data":"cc2706ef3964207f3b95d79eda84718d52480a4fed57c2232ac69803105bd452"} Aug 04 18:12:00.836316 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.836282 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" event={"ID":"b7384e93-e3d1-4ab3-bf64-1e932beac4f9","Type":"ContainerStarted","Data":"4038ec0ff0e46565742214d916b2c4e536f5f0b1c7e1aee06c6695df31f1af21"} Aug 04 18:12:00.836467 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.836338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" event={"ID":"b7384e93-e3d1-4ab3-bf64-1e932beac4f9","Type":"ContainerStarted","Data":"9586a6454314141d8dff6c6004eb3491ab6ded49122d99b5cfa5f2a7a409f6d6"} Aug 04 18:12:00.836467 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.836387 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" event={"ID":"b7384e93-e3d1-4ab3-bf64-1e932beac4f9","Type":"ContainerStarted","Data":"b1c3a959d621675ede1a881874dcbe117eb43292e5adb4ab65a8be07f17d2b57"} Aug 04 18:12:00.838530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.838499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" event={"ID":"2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5","Type":"ContainerStarted","Data":"5184efa5fe03db9ebf93e352c408a60bd765148e62b63e8c9b84b477662e79e6"} Aug 04 18:12:00.941982 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.941920 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-qhqvf" podStartSLOduration=2.468188106 podStartE2EDuration="3.941896936s" totalImagesPullingTime="1.47370883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:00.907085825 +0000 UTC m=+86.053902086" watchObservedRunningTime="2026-08-04 18:12:00.941896936 +0000 UTC m=+86.088713197" Aug 04 18:12:00.942763 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:00.942725 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-866wn" podStartSLOduration=2.360211084 podStartE2EDuration="3.942716535s" totalImagesPullingTime="1.582505451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:00.941874643 +0000 UTC m=+86.088690905" watchObservedRunningTime="2026-08-04 18:12:00.942716535 +0000 UTC m=+86.089532796" Aug 04 18:12:01.695242 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.695200 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dbc4959c4-qh68s"] Aug 04 18:12:01.698558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.698534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.701136 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.701113 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 18:12:01.701921 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.701897 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f8sq722u31eql\"" Aug 04 18:12:01.702063 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.701909 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 18:12:01.702063 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.701945 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 18:12:01.702063 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.701981 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 18:12:01.702278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.702259 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wn9lc\"" Aug 04 18:12:01.710572 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.710541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dbc4959c4-qh68s"] Aug 04 18:12:01.782764 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-client-ca-bundle\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.782947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/360f78b4-3925-4b2e-ac2d-2c948347762e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.782947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/360f78b4-3925-4b2e-ac2d-2c948347762e-audit-log\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.782947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8fw\" (UniqueName: \"kubernetes.io/projected/360f78b4-3925-4b2e-ac2d-2c948347762e-kube-api-access-nk8fw\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.782947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-secret-metrics-server-tls\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.782947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-secret-metrics-server-client-certs\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.782947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.782948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/360f78b4-3925-4b2e-ac2d-2c948347762e-metrics-server-audit-profiles\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.844901 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.844854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf6pt" event={"ID":"a8048dc7-8311-402c-a23e-1fcb4f629df8","Type":"ContainerStarted","Data":"fa9595dbb748ea2812e5f8d03a10658081d942689754b5156880d4b50dbd2b3d"} Aug 04 18:12:01.845385 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.844918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf6pt" event={"ID":"a8048dc7-8311-402c-a23e-1fcb4f629df8","Type":"ContainerStarted","Data":"9766ef4e6fc8b59809a0a9cc477e87123371fd281c1717d42bd8b59b0459349b"} Aug 04 18:12:01.879210 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.879148 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rf6pt" podStartSLOduration=3.006106723 podStartE2EDuration="4.87912959s" totalImagesPullingTime="1.77400975s" totalInitContainerRuntime="99.013117ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:01.875722767 +0000 UTC m=+87.022539045" watchObservedRunningTime="2026-08-04 18:12:01.87912959 +0000 UTC m=+87.025945850" Aug 04 18:12:01.883432 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-client-ca-bundle\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/360f78b4-3925-4b2e-ac2d-2c948347762e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/360f78b4-3925-4b2e-ac2d-2c948347762e-audit-log\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8fw\" (UniqueName: \"kubernetes.io/projected/360f78b4-3925-4b2e-ac2d-2c948347762e-kube-api-access-nk8fw\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-secret-metrics-server-tls\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-secret-metrics-server-client-certs\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883786 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/360f78b4-3925-4b2e-ac2d-2c948347762e-metrics-server-audit-profiles\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.883906 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.883882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/360f78b4-3925-4b2e-ac2d-2c948347762e-audit-log\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.884342 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.884314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/360f78b4-3925-4b2e-ac2d-2c948347762e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.884942 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.884915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/360f78b4-3925-4b2e-ac2d-2c948347762e-metrics-server-audit-profiles\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.886096 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.886068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-secret-metrics-server-client-certs\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.886174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.886130 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-client-ca-bundle\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.886174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.886161 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/360f78b4-3925-4b2e-ac2d-2c948347762e-secret-metrics-server-tls\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:01.897518 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:01.897495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8fw\" (UniqueName: \"kubernetes.io/projected/360f78b4-3925-4b2e-ac2d-2c948347762e-kube-api-access-nk8fw\") pod \"metrics-server-6dbc4959c4-qh68s\" (UID: \"360f78b4-3925-4b2e-ac2d-2c948347762e\") " pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:02.008382 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:02.008273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:02.155240 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:02.155206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dbc4959c4-qh68s"] Aug 04 18:12:02.158431 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:12:02.158404 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360f78b4_3925_4b2e_ac2d_2c948347762e.slice/crio-066d6e19083e05d4ec3ab05ca56c699d6fb77076b83991098d13f9eaf19f8cd1 WatchSource:0}: Error finding container 066d6e19083e05d4ec3ab05ca56c699d6fb77076b83991098d13f9eaf19f8cd1: Status 404 returned error can't find the container with id 066d6e19083e05d4ec3ab05ca56c699d6fb77076b83991098d13f9eaf19f8cd1 Aug 04 18:12:02.849521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:02.849482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" event={"ID":"360f78b4-3925-4b2e-ac2d-2c948347762e","Type":"ContainerStarted","Data":"066d6e19083e05d4ec3ab05ca56c699d6fb77076b83991098d13f9eaf19f8cd1"} Aug 04 18:12:03.074286 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.074250 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5649667b7d-dvbhd"] Aug 04 18:12:03.076578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.076561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.084043 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.083693 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 18:12:03.091129 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.091107 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5649667b7d-dvbhd"] Aug 04 18:12:03.195737 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-service-ca\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.195737 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-oauth-serving-cert\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.195916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-serving-cert\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.195916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-config\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.195916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrd2x\" (UniqueName: \"kubernetes.io/projected/38543420-ed70-4bc7-8d41-12d1b2e685c7-kube-api-access-hrd2x\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.195916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-oauth-config\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.195916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.195907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-trusted-ca-bundle\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-service-ca\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-oauth-serving-cert\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-serving-cert\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-config\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrd2x\" (UniqueName: \"kubernetes.io/projected/38543420-ed70-4bc7-8d41-12d1b2e685c7-kube-api-access-hrd2x\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-oauth-config\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.297390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.297284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-trusted-ca-bundle\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.298052 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.298022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-service-ca\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.298178 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.298063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-config\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.298178 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.298036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-oauth-serving-cert\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.298314 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.298292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-trusted-ca-bundle\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.299783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.299752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-oauth-config\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.299879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.299852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-serving-cert\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.309779 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.309758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrd2x\" (UniqueName: \"kubernetes.io/projected/38543420-ed70-4bc7-8d41-12d1b2e685c7-kube-api-access-hrd2x\") pod \"console-5649667b7d-dvbhd\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.386828 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.386790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:03.564584 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.564544 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5649667b7d-dvbhd"] Aug 04 18:12:03.568197 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:12:03.568172 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38543420_ed70_4bc7_8d41_12d1b2e685c7.slice/crio-7b326f67029e59a98649219bd3db5282fdf071c8cb07861b11de17cc7e7563e9 WatchSource:0}: Error finding container 7b326f67029e59a98649219bd3db5282fdf071c8cb07861b11de17cc7e7563e9: Status 404 returned error can't find the container with id 7b326f67029e59a98649219bd3db5282fdf071c8cb07861b11de17cc7e7563e9 Aug 04 18:12:03.751581 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.751500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:12:03.858739 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.858691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5649667b7d-dvbhd" event={"ID":"38543420-ed70-4bc7-8d41-12d1b2e685c7","Type":"ContainerStarted","Data":"13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e"} Aug 04 18:12:03.859136 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.858759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5649667b7d-dvbhd" event={"ID":"38543420-ed70-4bc7-8d41-12d1b2e685c7","Type":"ContainerStarted","Data":"7b326f67029e59a98649219bd3db5282fdf071c8cb07861b11de17cc7e7563e9"} Aug 04 18:12:03.883911 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:03.883857 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5649667b7d-dvbhd" podStartSLOduration=0.883837772 podStartE2EDuration="883.837772ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:03.882625289 +0000 UTC m=+89.029441550" watchObservedRunningTime="2026-08-04 18:12:03.883837772 +0000 UTC m=+89.030654036" Aug 04 18:12:04.864762 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:04.864720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" event={"ID":"360f78b4-3925-4b2e-ac2d-2c948347762e","Type":"ContainerStarted","Data":"70e60d881d25bef37e11dc1d346d0d48de95498d5b25659decd620cc8b061c22"} Aug 04 18:12:04.888352 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:04.888293 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" podStartSLOduration=1.9789352249999999 podStartE2EDuration="3.888275347s" totalImagesPullingTime="1.909340122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:04.88646365 +0000 UTC m=+90.033279911" watchObservedRunningTime="2026-08-04 18:12:04.888275347 +0000 UTC m=+90.035091607" Aug 04 18:12:04.991852 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:04.991813 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:12:04.992130 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:04.991874 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:12:04.997001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:04.996971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:12:05.872475 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:05.872444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:12:09.275074 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.275035 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5649667b7d-dvbhd"] Aug 04 18:12:09.319434 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.319397 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c8c97456f-sjxqr"] Aug 04 18:12:09.322767 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.322747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.334445 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.334419 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c8c97456f-sjxqr"] Aug 04 18:12:09.451608 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znb65\" (UniqueName: \"kubernetes.io/projected/0b1e976e-e7ec-4195-bf26-981e15f4848e-kube-api-access-znb65\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.451811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-service-ca\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.451811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-trusted-ca-bundle\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.451811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-serving-cert\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.451945 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-oauth-serving-cert\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.451945 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-oauth-config\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.451945 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.451873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-config\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.552795 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.552706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znb65\" (UniqueName: \"kubernetes.io/projected/0b1e976e-e7ec-4195-bf26-981e15f4848e-kube-api-access-znb65\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.552795 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.552754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-service-ca\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.552795 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.552777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-trusted-ca-bundle\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553066 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.552800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-serving-cert\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553066 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.552849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-oauth-serving-cert\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553066 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.552873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-oauth-config\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553066 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.553016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-config\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553681 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.553652 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-service-ca\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553681 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.553675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-oauth-serving-cert\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553884 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.553759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-config\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.553884 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.553837 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-trusted-ca-bundle\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.555905 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.555887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-oauth-config\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.556019 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.556003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-serving-cert\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.562792 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.562767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znb65\" (UniqueName: \"kubernetes.io/projected/0b1e976e-e7ec-4195-bf26-981e15f4848e-kube-api-access-znb65\") pod \"console-7c8c97456f-sjxqr\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.632704 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.632662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:09.768303 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.768264 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c8c97456f-sjxqr"] Aug 04 18:12:09.770773 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:12:09.770748 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1e976e_e7ec_4195_bf26_981e15f4848e.slice/crio-1a4b6788192462ab7a9bd107eab1470d22a4cbf525281ac12c021d0d31f13dbe WatchSource:0}: Error finding container 1a4b6788192462ab7a9bd107eab1470d22a4cbf525281ac12c021d0d31f13dbe: Status 404 returned error can't find the container with id 1a4b6788192462ab7a9bd107eab1470d22a4cbf525281ac12c021d0d31f13dbe Aug 04 18:12:09.887725 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.887685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8c97456f-sjxqr" event={"ID":"0b1e976e-e7ec-4195-bf26-981e15f4848e","Type":"ContainerStarted","Data":"9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5"} Aug 04 18:12:09.887878 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.887741 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8c97456f-sjxqr" event={"ID":"0b1e976e-e7ec-4195-bf26-981e15f4848e","Type":"ContainerStarted","Data":"1a4b6788192462ab7a9bd107eab1470d22a4cbf525281ac12c021d0d31f13dbe"} Aug 04 18:12:09.907753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:09.907689 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c8c97456f-sjxqr" podStartSLOduration=0.907672702 podStartE2EDuration="907.672702ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:09.90632924 +0000 UTC m=+95.053145501" watchObservedRunningTime="2026-08-04 18:12:09.907672702 +0000 UTC m=+95.054488961" Aug 04 18:12:10.567792 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:10.567759 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76fbdcb99f-qzxkm"] Aug 04 18:12:12.737760 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:12.737724 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fwqwx" Aug 04 18:12:13.387720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:13.387681 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:19.633626 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:19.633589 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:19.634021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:19.633653 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:19.638641 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:19.638616 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:19.928814 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:19.928784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:12:19.987288 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:19.987255 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d5574897-qvl5c"] Aug 04 18:12:22.008755 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:22.008713 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:22.009197 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:22.008785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:34.294801 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.294728 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5649667b7d-dvbhd" podUID="38543420-ed70-4bc7-8d41-12d1b2e685c7" containerName="console" containerID="cri-o://13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e" gracePeriod=15 Aug 04 18:12:34.574925 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.574888 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:34.654393 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654347 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-oauth-serving-cert\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.654393 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654398 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-config\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.654609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654438 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-trusted-ca-bundle\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.654609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654486 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-oauth-config\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.654726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654703 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-service-ca\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.654782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654760 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrd2x\" (UniqueName: \"kubernetes.io/projected/38543420-ed70-4bc7-8d41-12d1b2e685c7-kube-api-access-hrd2x\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.654831 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654771 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:34.655129 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654866 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-serving-cert\") pod \"38543420-ed70-4bc7-8d41-12d1b2e685c7\" (UID: \"38543420-ed70-4bc7-8d41-12d1b2e685c7\") " Aug 04 18:12:34.655129 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654867 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-config" (OuterVolumeSpecName: "console-config") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:34.655129 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.654965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:34.655129 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.655044 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:34.655334 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.655183 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-oauth-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.655334 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.655196 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.655334 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.655205 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-trusted-ca-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.655334 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.655218 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38543420-ed70-4bc7-8d41-12d1b2e685c7-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.656983 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.656950 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:34.657079 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.656961 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:34.657079 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.657018 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38543420-ed70-4bc7-8d41-12d1b2e685c7-kube-api-access-hrd2x" (OuterVolumeSpecName: "kube-api-access-hrd2x") pod "38543420-ed70-4bc7-8d41-12d1b2e685c7" (UID: "38543420-ed70-4bc7-8d41-12d1b2e685c7"). InnerVolumeSpecName "kube-api-access-hrd2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:34.756596 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.756551 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-oauth-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.756596 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.756590 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrd2x\" (UniqueName: \"kubernetes.io/projected/38543420-ed70-4bc7-8d41-12d1b2e685c7-kube-api-access-hrd2x\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.756596 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.756602 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38543420-ed70-4bc7-8d41-12d1b2e685c7-console-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:34.983463 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.983427 2575 generic.go:356] "Generic (PLEG): container finished" podID="38543420-ed70-4bc7-8d41-12d1b2e685c7" containerID="13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e" exitCode=0 Aug 04 18:12:34.983620 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.983478 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5649667b7d-dvbhd" Aug 04 18:12:34.983620 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.983500 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5649667b7d-dvbhd" event={"ID":"38543420-ed70-4bc7-8d41-12d1b2e685c7","Type":"ContainerDied","Data":"13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e"} Aug 04 18:12:34.983620 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.983548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5649667b7d-dvbhd" event={"ID":"38543420-ed70-4bc7-8d41-12d1b2e685c7","Type":"ContainerDied","Data":"7b326f67029e59a98649219bd3db5282fdf071c8cb07861b11de17cc7e7563e9"} Aug 04 18:12:34.983620 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.983574 2575 scope.go:117] "RemoveContainer" containerID="13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e" Aug 04 18:12:34.992674 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.992656 2575 scope.go:117] "RemoveContainer" containerID="13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e" Aug 04 18:12:34.992956 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:12:34.992937 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e\": container with ID starting with 13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e not found: ID does not exist" containerID="13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e" Aug 04 18:12:34.993003 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:34.992965 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e"} err="failed to get container status \"13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e\": rpc error: code = NotFound desc = could not find container \"13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e\": container with ID starting with 13c2777912344ca2c0c9c139719bd8c48f7659fb58390a991e590e836c1f407e not found: ID does not exist" Aug 04 18:12:35.009329 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.009304 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5649667b7d-dvbhd"] Aug 04 18:12:35.014346 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.014322 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5649667b7d-dvbhd"] Aug 04 18:12:35.347452 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.347414 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38543420-ed70-4bc7-8d41-12d1b2e685c7" path="/var/lib/kubelet/pods/38543420-ed70-4bc7-8d41-12d1b2e685c7/volumes" Aug 04 18:12:35.587964 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.587863 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" podUID="5af49ee1-5ad0-4f19-8765-0ae0c18acb08" containerName="registry" containerID="cri-o://05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423" gracePeriod=30 Aug 04 18:12:35.820992 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.820971 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:12:35.867634 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867568 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-bound-sa-token\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.867634 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867602 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n2bc\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-kube-api-access-9n2bc\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.867824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867652 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-certificates\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.867824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867684 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-image-registry-private-configuration\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.867824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867727 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.867824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867786 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-installation-pull-secrets\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.867824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-ca-trust-extracted\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.868083 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.867847 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-trusted-ca\") pod \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\" (UID: \"5af49ee1-5ad0-4f19-8765-0ae0c18acb08\") " Aug 04 18:12:35.868146 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.868100 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:35.868715 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.868686 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:35.870528 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.870500 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:35.870949 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.870924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-kube-api-access-9n2bc" (OuterVolumeSpecName: "kube-api-access-9n2bc") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "kube-api-access-9n2bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:35.871061 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.871035 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:35.871238 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.871212 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:35.871776 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.871739 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:35.878117 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.878092 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5af49ee1-5ad0-4f19-8765-0ae0c18acb08" (UID: "5af49ee1-5ad0-4f19-8765-0ae0c18acb08"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:12:35.969384 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969335 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-installation-pull-secrets\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969384 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969378 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-ca-trust-extracted\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969384 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969388 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-trusted-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969599 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969398 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-bound-sa-token\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969599 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969406 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2bc\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-kube-api-access-9n2bc\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969599 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969414 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-certificates\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969599 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969425 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-image-registry-private-configuration\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.969599 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.969434 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5af49ee1-5ad0-4f19-8765-0ae0c18acb08-registry-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:35.988753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.988722 2575 generic.go:356] "Generic (PLEG): container finished" podID="5af49ee1-5ad0-4f19-8765-0ae0c18acb08" containerID="05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423" exitCode=0 Aug 04 18:12:35.988877 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.988765 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" Aug 04 18:12:35.988877 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.988782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" event={"ID":"5af49ee1-5ad0-4f19-8765-0ae0c18acb08","Type":"ContainerDied","Data":"05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423"} Aug 04 18:12:35.988877 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.988826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76fbdcb99f-qzxkm" event={"ID":"5af49ee1-5ad0-4f19-8765-0ae0c18acb08","Type":"ContainerDied","Data":"983a7d1bd604fe605813e28693e03e02f103416f39852cf877e64ea6d0188896"} Aug 04 18:12:35.988877 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.988847 2575 scope.go:117] "RemoveContainer" containerID="05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423" Aug 04 18:12:35.997510 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.997488 2575 scope.go:117] "RemoveContainer" containerID="05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423" Aug 04 18:12:35.997784 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:12:35.997766 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423\": container with ID starting with 05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423 not found: ID does not exist" containerID="05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423" Aug 04 18:12:35.997840 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:35.997793 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423"} err="failed to get container status \"05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423\": rpc error: code = NotFound desc = could not find container \"05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423\": container with ID starting with 05068e9effa450ed6489ecf4d2b6e36aca8a5e3997572f7a9257c882bcbc9423 not found: ID does not exist" Aug 04 18:12:36.010749 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:36.010720 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76fbdcb99f-qzxkm"] Aug 04 18:12:36.019674 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:36.019652 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76fbdcb99f-qzxkm"] Aug 04 18:12:36.995006 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:36.994975 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc3a9444-8ab5-4152-aadd-c700368fc215" containerID="771a8e379ec0e5c5cbd4a18dbe8590656a082ae658e080f2005ee692b5eabc6a" exitCode=0 Aug 04 18:12:36.995520 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:36.995032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" event={"ID":"dc3a9444-8ab5-4152-aadd-c700368fc215","Type":"ContainerDied","Data":"771a8e379ec0e5c5cbd4a18dbe8590656a082ae658e080f2005ee692b5eabc6a"} Aug 04 18:12:36.995520 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:36.995455 2575 scope.go:117] "RemoveContainer" containerID="771a8e379ec0e5c5cbd4a18dbe8590656a082ae658e080f2005ee692b5eabc6a" Aug 04 18:12:37.348078 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:37.347984 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af49ee1-5ad0-4f19-8765-0ae0c18acb08" path="/var/lib/kubelet/pods/5af49ee1-5ad0-4f19-8765-0ae0c18acb08/volumes" Aug 04 18:12:38.001015 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:38.000974 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-std9x" event={"ID":"dc3a9444-8ab5-4152-aadd-c700368fc215","Type":"ContainerStarted","Data":"bebfe45c998437e3ba19c48dcedd67d51c7578772d3f9c90112ee0f95c4774e0"} Aug 04 18:12:42.013981 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:42.013950 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:42.017889 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:42.017867 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dbc4959c4-qh68s" Aug 04 18:12:45.007582 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.007514 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84d5574897-qvl5c" podUID="6df7a80a-6121-46d0-9e3c-f07e37eb6137" containerName="console" containerID="cri-o://3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a" gracePeriod=15 Aug 04 18:12:45.254039 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.254015 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:12:45.342934 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.342867 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-oauth-serving-cert\") pod \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " Aug 04 18:12:45.342934 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.342926 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-oauth-config\") pod \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " Aug 04 18:12:45.343138 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.342957 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dh5d\" (UniqueName: \"kubernetes.io/projected/6df7a80a-6121-46d0-9e3c-f07e37eb6137-kube-api-access-5dh5d\") pod \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " Aug 04 18:12:45.343138 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-config\") pod \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " Aug 04 18:12:45.343138 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343050 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-serving-cert\") pod \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " Aug 04 18:12:45.343138 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343073 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-service-ca\") pod \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\" (UID: \"6df7a80a-6121-46d0-9e3c-f07e37eb6137\") " Aug 04 18:12:45.343336 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343189 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6df7a80a-6121-46d0-9e3c-f07e37eb6137" (UID: "6df7a80a-6121-46d0-9e3c-f07e37eb6137"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:45.343336 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343266 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-oauth-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:45.343505 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343483 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-service-ca" (OuterVolumeSpecName: "service-ca") pod "6df7a80a-6121-46d0-9e3c-f07e37eb6137" (UID: "6df7a80a-6121-46d0-9e3c-f07e37eb6137"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:45.343876 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.343800 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-config" (OuterVolumeSpecName: "console-config") pod "6df7a80a-6121-46d0-9e3c-f07e37eb6137" (UID: "6df7a80a-6121-46d0-9e3c-f07e37eb6137"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:12:45.345922 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.345890 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6df7a80a-6121-46d0-9e3c-f07e37eb6137" (UID: "6df7a80a-6121-46d0-9e3c-f07e37eb6137"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:45.345922 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.345892 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df7a80a-6121-46d0-9e3c-f07e37eb6137-kube-api-access-5dh5d" (OuterVolumeSpecName: "kube-api-access-5dh5d") pod "6df7a80a-6121-46d0-9e3c-f07e37eb6137" (UID: "6df7a80a-6121-46d0-9e3c-f07e37eb6137"). InnerVolumeSpecName "kube-api-access-5dh5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:45.345922 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.345907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6df7a80a-6121-46d0-9e3c-f07e37eb6137" (UID: "6df7a80a-6121-46d0-9e3c-f07e37eb6137"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:12:45.444531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.444510 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-oauth-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:45.444531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.444532 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dh5d\" (UniqueName: \"kubernetes.io/projected/6df7a80a-6121-46d0-9e3c-f07e37eb6137-kube-api-access-5dh5d\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:45.444665 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.444542 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:45.444665 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.444550 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6df7a80a-6121-46d0-9e3c-f07e37eb6137-console-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:45.444665 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:45.444559 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6df7a80a-6121-46d0-9e3c-f07e37eb6137-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:12:46.033127 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.033092 2575 generic.go:356] "Generic (PLEG): container finished" podID="6df7a80a-6121-46d0-9e3c-f07e37eb6137" containerID="3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a" exitCode=0 Aug 04 18:12:46.033616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.033135 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d5574897-qvl5c" Aug 04 18:12:46.033616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.033183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d5574897-qvl5c" event={"ID":"6df7a80a-6121-46d0-9e3c-f07e37eb6137","Type":"ContainerDied","Data":"3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a"} Aug 04 18:12:46.033616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.033233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d5574897-qvl5c" event={"ID":"6df7a80a-6121-46d0-9e3c-f07e37eb6137","Type":"ContainerDied","Data":"9cde794a974191ceae42703420fdc3ad4f7efdd5f8d394c73d5151f6da28239f"} Aug 04 18:12:46.033616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.033256 2575 scope.go:117] "RemoveContainer" containerID="3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a" Aug 04 18:12:46.041500 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.041466 2575 scope.go:117] "RemoveContainer" containerID="3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a" Aug 04 18:12:46.041752 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:12:46.041735 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a\": container with ID starting with 3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a not found: ID does not exist" containerID="3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a" Aug 04 18:12:46.041820 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.041759 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a"} err="failed to get container status \"3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a\": rpc error: code = NotFound desc = could not find container \"3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a\": container with ID starting with 3ad81f8da7df272e08e6e615da25a7c43236932d4ee873c85d6a09f324387d6a not found: ID does not exist" Aug 04 18:12:46.052761 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.052736 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d5574897-qvl5c"] Aug 04 18:12:46.057661 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:46.057642 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d5574897-qvl5c"] Aug 04 18:12:47.038969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:47.038931 2575 generic.go:356] "Generic (PLEG): container finished" podID="b36af2a3-d7d0-469a-a78d-c20e9ffcb746" containerID="4454cef1aac2bfd032f4ba527496308c06ac22baef6f3f95d62f1ff51cb5a781" exitCode=0 Aug 04 18:12:47.039467 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:47.039007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-m7zkx" event={"ID":"b36af2a3-d7d0-469a-a78d-c20e9ffcb746","Type":"ContainerDied","Data":"4454cef1aac2bfd032f4ba527496308c06ac22baef6f3f95d62f1ff51cb5a781"} Aug 04 18:12:47.039548 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:47.039486 2575 scope.go:117] "RemoveContainer" containerID="4454cef1aac2bfd032f4ba527496308c06ac22baef6f3f95d62f1ff51cb5a781" Aug 04 18:12:47.348306 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:47.348216 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df7a80a-6121-46d0-9e3c-f07e37eb6137" path="/var/lib/kubelet/pods/6df7a80a-6121-46d0-9e3c-f07e37eb6137/volumes" Aug 04 18:12:48.045033 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:48.044995 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-m7zkx" event={"ID":"b36af2a3-d7d0-469a-a78d-c20e9ffcb746","Type":"ContainerStarted","Data":"bbc73131c47389b716c210453010f4f0481d90ff40fc91f5311e958c9cf026a2"} Aug 04 18:12:52.060943 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:52.060912 2575 generic.go:356] "Generic (PLEG): container finished" podID="f8d6a004-71d5-46dc-b7bb-06bfa86f8131" containerID="1521f24ca014c4f78c17cc5b322ac15686c29c4b142480b8dce4e3cdf14fb4a2" exitCode=0 Aug 04 18:12:52.061399 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:52.060992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" event={"ID":"f8d6a004-71d5-46dc-b7bb-06bfa86f8131","Type":"ContainerDied","Data":"1521f24ca014c4f78c17cc5b322ac15686c29c4b142480b8dce4e3cdf14fb4a2"} Aug 04 18:12:52.061399 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:52.061316 2575 scope.go:117] "RemoveContainer" containerID="1521f24ca014c4f78c17cc5b322ac15686c29c4b142480b8dce4e3cdf14fb4a2" Aug 04 18:12:53.067298 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:12:53.067262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-v666v" event={"ID":"f8d6a004-71d5-46dc-b7bb-06bfa86f8131","Type":"ContainerStarted","Data":"9a795199f18ae60455cfaefde340788c3cbc0b3660c7cb03e18d54fb5443b4ac"} Aug 04 18:13:16.266108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266069 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc584c66d-6bjtf"] Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266354 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df7a80a-6121-46d0-9e3c-f07e37eb6137" containerName="console" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266384 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df7a80a-6121-46d0-9e3c-f07e37eb6137" containerName="console" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266400 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38543420-ed70-4bc7-8d41-12d1b2e685c7" containerName="console" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266405 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38543420-ed70-4bc7-8d41-12d1b2e685c7" containerName="console" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266412 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af49ee1-5ad0-4f19-8765-0ae0c18acb08" containerName="registry" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266417 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af49ee1-5ad0-4f19-8765-0ae0c18acb08" containerName="registry" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266478 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af49ee1-5ad0-4f19-8765-0ae0c18acb08" containerName="registry" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266486 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df7a80a-6121-46d0-9e3c-f07e37eb6137" containerName="console" Aug 04 18:13:16.266553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.266493 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38543420-ed70-4bc7-8d41-12d1b2e685c7" containerName="console" Aug 04 18:13:16.273283 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.271152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.282823 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.282796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc584c66d-6bjtf"] Aug 04 18:13:16.306842 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.306812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-trusted-ca-bundle\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.306842 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.306845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-config\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.307021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.306893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-serving-cert\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.307021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.306920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-oauth-serving-cert\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.307021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.306982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjng\" (UniqueName: \"kubernetes.io/projected/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-kube-api-access-4kjng\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.307021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.307002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-oauth-config\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.307021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.307017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-service-ca\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.407622 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-trusted-ca-bundle\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.407622 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-config\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.407844 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-serving-cert\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.407844 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-oauth-serving-cert\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.407844 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjng\" (UniqueName: \"kubernetes.io/projected/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-kube-api-access-4kjng\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.407844 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-oauth-config\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.408003 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.407850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-service-ca\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.408556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.408527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-config\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.408684 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.408558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-trusted-ca-bundle\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.408684 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.408527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-oauth-serving-cert\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.408777 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.408677 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-service-ca\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.410460 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.410432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-serving-cert\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.410566 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.410527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-oauth-config\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.424623 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.424596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjng\" (UniqueName: \"kubernetes.io/projected/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-kube-api-access-4kjng\") pod \"console-dc584c66d-6bjtf\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.582031 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.581946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:16.717437 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:16.717395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc584c66d-6bjtf"] Aug 04 18:13:16.719482 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:13:16.719453 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac50e80e_bcb2_4145_8665_f5c7a4d77a41.slice/crio-a06704606607113baff99a7724a2b6aef65bde62b07e334dcabbf2db1c43061e WatchSource:0}: Error finding container a06704606607113baff99a7724a2b6aef65bde62b07e334dcabbf2db1c43061e: Status 404 returned error can't find the container with id a06704606607113baff99a7724a2b6aef65bde62b07e334dcabbf2db1c43061e Aug 04 18:13:17.163473 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:17.163427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc584c66d-6bjtf" event={"ID":"ac50e80e-bcb2-4145-8665-f5c7a4d77a41","Type":"ContainerStarted","Data":"eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5"} Aug 04 18:13:17.163473 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:17.163479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc584c66d-6bjtf" event={"ID":"ac50e80e-bcb2-4145-8665-f5c7a4d77a41","Type":"ContainerStarted","Data":"a06704606607113baff99a7724a2b6aef65bde62b07e334dcabbf2db1c43061e"} Aug 04 18:13:17.198849 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:17.198793 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dc584c66d-6bjtf" podStartSLOduration=1.1987756059999999 podStartE2EDuration="1.198775606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:17.198231922 +0000 UTC m=+162.345048183" watchObservedRunningTime="2026-08-04 18:13:17.198775606 +0000 UTC m=+162.345591860" Aug 04 18:13:26.582819 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:26.582731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:26.583225 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:26.583012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:26.587769 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:26.587747 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:27.207034 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:27.207004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:13:27.259968 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:27.259922 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c8c97456f-sjxqr"] Aug 04 18:13:52.281960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.281892 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c8c97456f-sjxqr" podUID="0b1e976e-e7ec-4195-bf26-981e15f4848e" containerName="console" containerID="cri-o://9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5" gracePeriod=15 Aug 04 18:13:52.525641 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.525616 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:13:52.591554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-oauth-config\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.591554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591515 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-trusted-ca-bundle\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.591554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591548 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-config\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.591872 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591581 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-service-ca\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.591872 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591609 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znb65\" (UniqueName: \"kubernetes.io/projected/0b1e976e-e7ec-4195-bf26-981e15f4848e-kube-api-access-znb65\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.591872 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591645 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-oauth-serving-cert\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.591872 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.591737 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-serving-cert\") pod \"0b1e976e-e7ec-4195-bf26-981e15f4848e\" (UID: \"0b1e976e-e7ec-4195-bf26-981e15f4848e\") " Aug 04 18:13:52.592083 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.592051 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-config" (OuterVolumeSpecName: "console-config") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:13:52.592155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.592077 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:13:52.592155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.592122 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:13:52.592384 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.592309 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-service-ca" (OuterVolumeSpecName: "service-ca") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:13:52.593852 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.593827 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:13:52.593971 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.593947 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1e976e-e7ec-4195-bf26-981e15f4848e-kube-api-access-znb65" (OuterVolumeSpecName: "kube-api-access-znb65") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "kube-api-access-znb65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:13:52.594040 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.594018 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0b1e976e-e7ec-4195-bf26-981e15f4848e" (UID: "0b1e976e-e7ec-4195-bf26-981e15f4848e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:13:52.692768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692724 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:52.692768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692762 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-oauth-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:52.692768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692775 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-trusted-ca-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:52.693010 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692788 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-console-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:52.693010 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692800 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:52.693010 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692812 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znb65\" (UniqueName: \"kubernetes.io/projected/0b1e976e-e7ec-4195-bf26-981e15f4848e-kube-api-access-znb65\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:52.693010 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:52.692824 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b1e976e-e7ec-4195-bf26-981e15f4848e-oauth-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:13:53.301728 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.301693 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b1e976e-e7ec-4195-bf26-981e15f4848e" containerID="9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5" exitCode=0 Aug 04 18:13:53.302147 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.301744 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8c97456f-sjxqr" Aug 04 18:13:53.302147 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.301750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8c97456f-sjxqr" event={"ID":"0b1e976e-e7ec-4195-bf26-981e15f4848e","Type":"ContainerDied","Data":"9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5"} Aug 04 18:13:53.302147 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.301789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8c97456f-sjxqr" event={"ID":"0b1e976e-e7ec-4195-bf26-981e15f4848e","Type":"ContainerDied","Data":"1a4b6788192462ab7a9bd107eab1470d22a4cbf525281ac12c021d0d31f13dbe"} Aug 04 18:13:53.302147 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.301808 2575 scope.go:117] "RemoveContainer" containerID="9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5" Aug 04 18:13:53.310471 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.310446 2575 scope.go:117] "RemoveContainer" containerID="9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5" Aug 04 18:13:53.310754 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:13:53.310734 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5\": container with ID starting with 9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5 not found: ID does not exist" containerID="9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5" Aug 04 18:13:53.310816 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.310764 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5"} err="failed to get container status \"9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5\": rpc error: code = NotFound desc = could not find container \"9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5\": container with ID starting with 9ce8cfa8c8b4caeab461c31db6e5c0ca59adc7f45425fb072af1fc3329c56dc5 not found: ID does not exist" Aug 04 18:13:53.321740 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.321710 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c8c97456f-sjxqr"] Aug 04 18:13:53.326042 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.326018 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c8c97456f-sjxqr"] Aug 04 18:13:53.347686 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:13:53.347656 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1e976e-e7ec-4195-bf26-981e15f4848e" path="/var/lib/kubelet/pods/0b1e976e-e7ec-4195-bf26-981e15f4848e/volumes" Aug 04 18:15:35.249926 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:35.249897 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:15:35.253664 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:35.253642 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:15:35.261580 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:35.261558 2575 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 18:15:48.524929 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.524895 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2"] Aug 04 18:15:48.527295 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.525209 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1e976e-e7ec-4195-bf26-981e15f4848e" containerName="console" Aug 04 18:15:48.527295 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.525221 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1e976e-e7ec-4195-bf26-981e15f4848e" containerName="console" Aug 04 18:15:48.527295 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.525285 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1e976e-e7ec-4195-bf26-981e15f4848e" containerName="console" Aug 04 18:15:48.528181 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.528165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.533776 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.533751 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 18:15:48.533903 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.533850 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6bz9z\"" Aug 04 18:15:48.534652 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.534630 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 18:15:48.540083 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.540063 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2"] Aug 04 18:15:48.656391 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.656314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.656590 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.656449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdn7z\" (UniqueName: \"kubernetes.io/projected/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-kube-api-access-hdn7z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.656590 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.656485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.757881 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.757850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.758065 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.757901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdn7z\" (UniqueName: \"kubernetes.io/projected/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-kube-api-access-hdn7z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.758065 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.757923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.758237 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.758220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.758302 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.758277 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.767622 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.767593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdn7z\" (UniqueName: \"kubernetes.io/projected/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-kube-api-access-hdn7z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.837454 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.837391 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:15:48.971622 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.971589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2"] Aug 04 18:15:48.975760 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:15:48.975733 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13c8dd9_cdf9_4b84_8781_9b1bd7e422ad.slice/crio-eba0b88d141dcc3c4c10b1662b704ba0f315ce3ef08d93f037ed3071e289fbd1 WatchSource:0}: Error finding container eba0b88d141dcc3c4c10b1662b704ba0f315ce3ef08d93f037ed3071e289fbd1: Status 404 returned error can't find the container with id eba0b88d141dcc3c4c10b1662b704ba0f315ce3ef08d93f037ed3071e289fbd1 Aug 04 18:15:48.978507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:48.978487 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:15:49.744167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:49.744124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" event={"ID":"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad","Type":"ContainerStarted","Data":"eba0b88d141dcc3c4c10b1662b704ba0f315ce3ef08d93f037ed3071e289fbd1"} Aug 04 18:15:56.778759 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:56.778718 2575 generic.go:356] "Generic (PLEG): container finished" podID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerID="7dcb43d11aff87277e6d1df0011caf0c0085b9fbaa34fdda80c9c8dfcd52af0f" exitCode=0 Aug 04 18:15:56.779180 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:56.778810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" event={"ID":"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad","Type":"ContainerDied","Data":"7dcb43d11aff87277e6d1df0011caf0c0085b9fbaa34fdda80c9c8dfcd52af0f"} Aug 04 18:15:59.792202 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:59.792161 2575 generic.go:356] "Generic (PLEG): container finished" podID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerID="e4c00df592bac478910673a1752fd43ca0ca74360f4ca99bae709e03f0edab2d" exitCode=0 Aug 04 18:15:59.792586 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:15:59.792236 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" event={"ID":"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad","Type":"ContainerDied","Data":"e4c00df592bac478910673a1752fd43ca0ca74360f4ca99bae709e03f0edab2d"} Aug 04 18:16:07.829004 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:07.828966 2575 generic.go:356] "Generic (PLEG): container finished" podID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerID="3c321f3ce7dc28bb85d561de87658eb6417b5445072582af973791a8718cb564" exitCode=0 Aug 04 18:16:07.829395 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:07.829032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" event={"ID":"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad","Type":"ContainerDied","Data":"3c321f3ce7dc28bb85d561de87658eb6417b5445072582af973791a8718cb564"} Aug 04 18:16:08.963712 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:08.963686 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:16:09.029341 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.029306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-bundle\") pod \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " Aug 04 18:16:09.029516 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.029414 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdn7z\" (UniqueName: \"kubernetes.io/projected/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-kube-api-access-hdn7z\") pod \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " Aug 04 18:16:09.029516 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.029432 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-util\") pod \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\" (UID: \"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad\") " Aug 04 18:16:09.030030 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.029997 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-bundle" (OuterVolumeSpecName: "bundle") pod "e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" (UID: "e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:16:09.031720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.031691 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-kube-api-access-hdn7z" (OuterVolumeSpecName: "kube-api-access-hdn7z") pod "e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" (UID: "e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad"). InnerVolumeSpecName "kube-api-access-hdn7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:16:09.035999 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.035971 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-util" (OuterVolumeSpecName: "util") pod "e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" (UID: "e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:16:09.130760 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.130675 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:09.130760 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.130707 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdn7z\" (UniqueName: \"kubernetes.io/projected/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-kube-api-access-hdn7z\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:09.130760 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.130718 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:09.838950 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.838918 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" Aug 04 18:16:09.839134 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.838917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b9sj2" event={"ID":"e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad","Type":"ContainerDied","Data":"eba0b88d141dcc3c4c10b1662b704ba0f315ce3ef08d93f037ed3071e289fbd1"} Aug 04 18:16:09.839134 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:09.839043 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba0b88d141dcc3c4c10b1662b704ba0f315ce3ef08d93f037ed3071e289fbd1" Aug 04 18:16:16.793922 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.793873 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz"] Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794164 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="extract" Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794175 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="extract" Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794184 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="util" Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794190 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="util" Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794211 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="pull" Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794216 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="pull" Aug 04 18:16:16.794339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.794263 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13c8dd9-cdf9-4b84-8781-9b1bd7e422ad" containerName="extract" Aug 04 18:16:16.798518 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.798499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:16.801154 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.801133 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hj66p\"" Aug 04 18:16:16.801273 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.801204 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 18:16:16.801273 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.801214 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:16:16.813866 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.813845 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz"] Aug 04 18:16:16.894515 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.894489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kj5\" (UniqueName: \"kubernetes.io/projected/2c2158c8-c3b1-49ac-98ea-a65c44675793-kube-api-access-76kj5\") pod \"cert-manager-operator-controller-manager-55c444bf86-6zppz\" (UID: \"2c2158c8-c3b1-49ac-98ea-a65c44675793\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:16.894691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.894603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c2158c8-c3b1-49ac-98ea-a65c44675793-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6zppz\" (UID: \"2c2158c8-c3b1-49ac-98ea-a65c44675793\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:16.995684 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.995648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c2158c8-c3b1-49ac-98ea-a65c44675793-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6zppz\" (UID: \"2c2158c8-c3b1-49ac-98ea-a65c44675793\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:16.995849 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.995703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kj5\" (UniqueName: \"kubernetes.io/projected/2c2158c8-c3b1-49ac-98ea-a65c44675793-kube-api-access-76kj5\") pod \"cert-manager-operator-controller-manager-55c444bf86-6zppz\" (UID: \"2c2158c8-c3b1-49ac-98ea-a65c44675793\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:16.996063 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:16.996042 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c2158c8-c3b1-49ac-98ea-a65c44675793-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6zppz\" (UID: \"2c2158c8-c3b1-49ac-98ea-a65c44675793\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:17.005718 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:17.005693 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kj5\" (UniqueName: \"kubernetes.io/projected/2c2158c8-c3b1-49ac-98ea-a65c44675793-kube-api-access-76kj5\") pod \"cert-manager-operator-controller-manager-55c444bf86-6zppz\" (UID: \"2c2158c8-c3b1-49ac-98ea-a65c44675793\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:17.107844 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:17.107761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" Aug 04 18:16:17.258990 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:17.258953 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz"] Aug 04 18:16:17.264339 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:16:17.264304 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2158c8_c3b1_49ac_98ea_a65c44675793.slice/crio-3246e02df13cf63e7d22633a97ecdea0a5efb41d5bcf576122f2ce45d41aa802 WatchSource:0}: Error finding container 3246e02df13cf63e7d22633a97ecdea0a5efb41d5bcf576122f2ce45d41aa802: Status 404 returned error can't find the container with id 3246e02df13cf63e7d22633a97ecdea0a5efb41d5bcf576122f2ce45d41aa802 Aug 04 18:16:17.871849 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:17.871806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" event={"ID":"2c2158c8-c3b1-49ac-98ea-a65c44675793","Type":"ContainerStarted","Data":"3246e02df13cf63e7d22633a97ecdea0a5efb41d5bcf576122f2ce45d41aa802"} Aug 04 18:16:19.680580 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.680546 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4l9mt"] Aug 04 18:16:19.682854 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.682834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.687207 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.687186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 18:16:19.687392 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.687349 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 18:16:19.688435 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.688419 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zzl7r\"" Aug 04 18:16:19.694999 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.694976 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4l9mt"] Aug 04 18:16:19.721532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.721507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5q2\" (UniqueName: \"kubernetes.io/projected/6638e3a7-b808-4de6-bcdd-2a60d9b0005b-kube-api-access-nm5q2\") pod \"cert-manager-webhook-6486ffd997-4l9mt\" (UID: \"6638e3a7-b808-4de6-bcdd-2a60d9b0005b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.721664 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.721563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6638e3a7-b808-4de6-bcdd-2a60d9b0005b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4l9mt\" (UID: \"6638e3a7-b808-4de6-bcdd-2a60d9b0005b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.822856 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.822806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5q2\" (UniqueName: \"kubernetes.io/projected/6638e3a7-b808-4de6-bcdd-2a60d9b0005b-kube-api-access-nm5q2\") pod \"cert-manager-webhook-6486ffd997-4l9mt\" (UID: \"6638e3a7-b808-4de6-bcdd-2a60d9b0005b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.823083 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.822874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6638e3a7-b808-4de6-bcdd-2a60d9b0005b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4l9mt\" (UID: \"6638e3a7-b808-4de6-bcdd-2a60d9b0005b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.831998 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.831963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6638e3a7-b808-4de6-bcdd-2a60d9b0005b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4l9mt\" (UID: \"6638e3a7-b808-4de6-bcdd-2a60d9b0005b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.832119 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.832091 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5q2\" (UniqueName: \"kubernetes.io/projected/6638e3a7-b808-4de6-bcdd-2a60d9b0005b-kube-api-access-nm5q2\") pod \"cert-manager-webhook-6486ffd997-4l9mt\" (UID: \"6638e3a7-b808-4de6-bcdd-2a60d9b0005b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:19.883416 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.883351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" event={"ID":"2c2158c8-c3b1-49ac-98ea-a65c44675793","Type":"ContainerStarted","Data":"05a798e9b7bcffb51eee8611dc0330c41266d202c2b7bf34f8875c63530d0d31"} Aug 04 18:16:19.903891 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:19.903840 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6zppz" podStartSLOduration=2.245761531 podStartE2EDuration="3.903822882s" totalImagesPullingTime="1.658061351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:19.90301892 +0000 UTC m=+345.049835177" watchObservedRunningTime="2026-08-04 18:16:19.903822882 +0000 UTC m=+345.050639143" Aug 04 18:16:20.004310 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.004212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:20.145965 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.145925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4l9mt"] Aug 04 18:16:20.148943 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:16:20.148914 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6638e3a7_b808_4de6_bcdd_2a60d9b0005b.slice/crio-366bfc062044f5c0abaf09fb98f7c2b4f53ec4bae11fe2dd58e8ba97d95dd76e WatchSource:0}: Error finding container 366bfc062044f5c0abaf09fb98f7c2b4f53ec4bae11fe2dd58e8ba97d95dd76e: Status 404 returned error can't find the container with id 366bfc062044f5c0abaf09fb98f7c2b4f53ec4bae11fe2dd58e8ba97d95dd76e Aug 04 18:16:20.816481 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.816445 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs"] Aug 04 18:16:20.819605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.819588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:20.826662 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.826636 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 18:16:20.827660 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.827643 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6bz9z\"" Aug 04 18:16:20.827660 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.827654 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 18:16:20.836986 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.836962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs"] Aug 04 18:16:20.888832 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.888802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" event={"ID":"6638e3a7-b808-4de6-bcdd-2a60d9b0005b","Type":"ContainerStarted","Data":"366bfc062044f5c0abaf09fb98f7c2b4f53ec4bae11fe2dd58e8ba97d95dd76e"} Aug 04 18:16:20.930605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.930566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:20.930605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.930603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6tz\" (UniqueName: \"kubernetes.io/projected/d3dd994f-23a1-4156-ae27-a1f63cc2101b-kube-api-access-7t6tz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:20.930819 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:20.930630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.031785 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.031741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.031967 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.031795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6tz\" (UniqueName: \"kubernetes.io/projected/d3dd994f-23a1-4156-ae27-a1f63cc2101b-kube-api-access-7t6tz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.031967 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.031839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.032245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.032220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.032324 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.032243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.044896 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.044869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6tz\" (UniqueName: \"kubernetes.io/projected/d3dd994f-23a1-4156-ae27-a1f63cc2101b-kube-api-access-7t6tz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.129787 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.129702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:21.288516 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.288489 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs"] Aug 04 18:16:21.291030 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:16:21.290991 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dd994f_23a1_4156_ae27_a1f63cc2101b.slice/crio-26408bf379d73e3e2cfab2641a4ac1598b83ce1827e615b474ab4ddbb9a3df3f WatchSource:0}: Error finding container 26408bf379d73e3e2cfab2641a4ac1598b83ce1827e615b474ab4ddbb9a3df3f: Status 404 returned error can't find the container with id 26408bf379d73e3e2cfab2641a4ac1598b83ce1827e615b474ab4ddbb9a3df3f Aug 04 18:16:21.896928 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.896836 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerID="2b4a59dadff4b706fee06419a061ea8f71d2e6f988966c39a6b170312e19251d" exitCode=0 Aug 04 18:16:21.896928 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.896911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" event={"ID":"d3dd994f-23a1-4156-ae27-a1f63cc2101b","Type":"ContainerDied","Data":"2b4a59dadff4b706fee06419a061ea8f71d2e6f988966c39a6b170312e19251d"} Aug 04 18:16:21.897469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:21.896963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" event={"ID":"d3dd994f-23a1-4156-ae27-a1f63cc2101b","Type":"ContainerStarted","Data":"26408bf379d73e3e2cfab2641a4ac1598b83ce1827e615b474ab4ddbb9a3df3f"} Aug 04 18:16:23.908528 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:23.908473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" event={"ID":"6638e3a7-b808-4de6-bcdd-2a60d9b0005b","Type":"ContainerStarted","Data":"92eb82ebd07d0af0a676be8fbcfc862b0ea0dda1238c9e898c5c804fa79e5ce6"} Aug 04 18:16:23.908971 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:23.908550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:23.928678 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:23.928626 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" podStartSLOduration=2.024357512 podStartE2EDuration="4.928609345s" totalImagesPullingTime="2.904251833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:23.927690293 +0000 UTC m=+349.074506554" watchObservedRunningTime="2026-08-04 18:16:23.928609345 +0000 UTC m=+349.075425606" Aug 04 18:16:24.915558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:24.915475 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerID="daa5734d9261db2f566c4ce7cd3a43aa49bfbd5536c7a87c72c6690a994b6943" exitCode=0 Aug 04 18:16:24.915932 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:24.915562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" event={"ID":"d3dd994f-23a1-4156-ae27-a1f63cc2101b","Type":"ContainerDied","Data":"daa5734d9261db2f566c4ce7cd3a43aa49bfbd5536c7a87c72c6690a994b6943"} Aug 04 18:16:25.922381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:25.922288 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerID="c56c73dcbd7f18579a341b3301fb9698a04bca622fba98404d9cce2ceb6265e5" exitCode=0 Aug 04 18:16:25.922727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:25.922382 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" event={"ID":"d3dd994f-23a1-4156-ae27-a1f63cc2101b","Type":"ContainerDied","Data":"c56c73dcbd7f18579a341b3301fb9698a04bca622fba98404d9cce2ceb6265e5"} Aug 04 18:16:26.535678 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.535644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl"] Aug 04 18:16:26.539249 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.539233 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.542726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.542705 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mtvtd\"" Aug 04 18:16:26.552594 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.552562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl"] Aug 04 18:16:26.584032 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.583994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6eef80ba-cb35-4d8e-8253-22d54c6bf371-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6nxsl\" (UID: \"6eef80ba-cb35-4d8e-8253-22d54c6bf371\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.584194 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.584146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7dh\" (UniqueName: \"kubernetes.io/projected/6eef80ba-cb35-4d8e-8253-22d54c6bf371-kube-api-access-8p7dh\") pod \"cert-manager-cainjector-559cdd8c87-6nxsl\" (UID: \"6eef80ba-cb35-4d8e-8253-22d54c6bf371\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.685342 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.685300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7dh\" (UniqueName: \"kubernetes.io/projected/6eef80ba-cb35-4d8e-8253-22d54c6bf371-kube-api-access-8p7dh\") pod \"cert-manager-cainjector-559cdd8c87-6nxsl\" (UID: \"6eef80ba-cb35-4d8e-8253-22d54c6bf371\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.685578 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.685357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6eef80ba-cb35-4d8e-8253-22d54c6bf371-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6nxsl\" (UID: \"6eef80ba-cb35-4d8e-8253-22d54c6bf371\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.695306 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.695269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7dh\" (UniqueName: \"kubernetes.io/projected/6eef80ba-cb35-4d8e-8253-22d54c6bf371-kube-api-access-8p7dh\") pod \"cert-manager-cainjector-559cdd8c87-6nxsl\" (UID: \"6eef80ba-cb35-4d8e-8253-22d54c6bf371\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.696870 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.696845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6eef80ba-cb35-4d8e-8253-22d54c6bf371-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6nxsl\" (UID: \"6eef80ba-cb35-4d8e-8253-22d54c6bf371\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.848065 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.847978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" Aug 04 18:16:26.989124 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:26.989007 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl"] Aug 04 18:16:26.993571 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:16:26.993543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eef80ba_cb35_4d8e_8253_22d54c6bf371.slice/crio-5f56b8e6cc37c37736a6ff1cc0008ed00a5d82811c601fda06bf0b47d594325e WatchSource:0}: Error finding container 5f56b8e6cc37c37736a6ff1cc0008ed00a5d82811c601fda06bf0b47d594325e: Status 404 returned error can't find the container with id 5f56b8e6cc37c37736a6ff1cc0008ed00a5d82811c601fda06bf0b47d594325e Aug 04 18:16:27.047321 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.047297 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:27.088691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.088620 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-bundle\") pod \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " Aug 04 18:16:27.088920 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.088891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-bundle" (OuterVolumeSpecName: "bundle") pod "d3dd994f-23a1-4156-ae27-a1f63cc2101b" (UID: "d3dd994f-23a1-4156-ae27-a1f63cc2101b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:16:27.089236 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.089216 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t6tz\" (UniqueName: \"kubernetes.io/projected/d3dd994f-23a1-4156-ae27-a1f63cc2101b-kube-api-access-7t6tz\") pod \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " Aug 04 18:16:27.089287 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.089274 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-util\") pod \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\" (UID: \"d3dd994f-23a1-4156-ae27-a1f63cc2101b\") " Aug 04 18:16:27.089598 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.089579 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:27.091134 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.091108 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dd994f-23a1-4156-ae27-a1f63cc2101b-kube-api-access-7t6tz" (OuterVolumeSpecName: "kube-api-access-7t6tz") pod "d3dd994f-23a1-4156-ae27-a1f63cc2101b" (UID: "d3dd994f-23a1-4156-ae27-a1f63cc2101b"). InnerVolumeSpecName "kube-api-access-7t6tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:16:27.093926 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.093905 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-util" (OuterVolumeSpecName: "util") pod "d3dd994f-23a1-4156-ae27-a1f63cc2101b" (UID: "d3dd994f-23a1-4156-ae27-a1f63cc2101b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:16:27.190108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.190074 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6tz\" (UniqueName: \"kubernetes.io/projected/d3dd994f-23a1-4156-ae27-a1f63cc2101b-kube-api-access-7t6tz\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:27.190108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.190101 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3dd994f-23a1-4156-ae27-a1f63cc2101b-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:27.933338 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.933307 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" Aug 04 18:16:27.933495 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.933308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwx6xs" event={"ID":"d3dd994f-23a1-4156-ae27-a1f63cc2101b","Type":"ContainerDied","Data":"26408bf379d73e3e2cfab2641a4ac1598b83ce1827e615b474ab4ddbb9a3df3f"} Aug 04 18:16:27.933495 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.933415 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26408bf379d73e3e2cfab2641a4ac1598b83ce1827e615b474ab4ddbb9a3df3f" Aug 04 18:16:27.934932 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.934904 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" event={"ID":"6eef80ba-cb35-4d8e-8253-22d54c6bf371","Type":"ContainerStarted","Data":"dba60bc12d7a699f429fbb27f847594d33f184e77d19596bed5894a5b0e88684"} Aug 04 18:16:27.935060 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.934942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" event={"ID":"6eef80ba-cb35-4d8e-8253-22d54c6bf371","Type":"ContainerStarted","Data":"5f56b8e6cc37c37736a6ff1cc0008ed00a5d82811c601fda06bf0b47d594325e"} Aug 04 18:16:27.959061 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:27.958973 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6nxsl" podStartSLOduration=1.958960722 podStartE2EDuration="1.958960722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:27.957515754 +0000 UTC m=+353.104332029" watchObservedRunningTime="2026-08-04 18:16:27.958960722 +0000 UTC m=+353.105776981" Aug 04 18:16:29.918666 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:29.918632 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4l9mt" Aug 04 18:16:47.857012 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.856976 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7"] Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857280 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="pull" Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857293 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="pull" Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857309 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="util" Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857315 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="util" Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857322 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="extract" Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857327 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="extract" Aug 04 18:16:47.857595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.857421 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3dd994f-23a1-4156-ae27-a1f63cc2101b" containerName="extract" Aug 04 18:16:47.861902 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.861880 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:47.864404 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.864380 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6bz9z\"" Aug 04 18:16:47.864771 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.864751 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 18:16:47.865193 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.865176 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 18:16:47.869566 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.869545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7"] Aug 04 18:16:47.970644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.970611 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:47.970812 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.970653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdjh\" (UniqueName: \"kubernetes.io/projected/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-kube-api-access-9rdjh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:47.970812 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:47.970733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.071515 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.071481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdjh\" (UniqueName: \"kubernetes.io/projected/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-kube-api-access-9rdjh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.071685 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.071548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.071685 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.071628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.071980 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.071961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.072021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.071983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.081032 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.081004 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdjh\" (UniqueName: \"kubernetes.io/projected/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-kube-api-access-9rdjh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.172833 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.172805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:48.308537 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:48.308490 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7"] Aug 04 18:16:48.311337 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:16:48.311311 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c35d25d_7da3_4ba5_bb5e_937c97e525cf.slice/crio-f4b90d98805cd1c899499af62bfab03bd2d555580ec9082ce9032e2caaf6e1db WatchSource:0}: Error finding container f4b90d98805cd1c899499af62bfab03bd2d555580ec9082ce9032e2caaf6e1db: Status 404 returned error can't find the container with id f4b90d98805cd1c899499af62bfab03bd2d555580ec9082ce9032e2caaf6e1db Aug 04 18:16:49.028403 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:49.028344 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerID="11dd5349a2f690f9aedd9466190b0b34a79eade57d6162a32468b3175456583c" exitCode=0 Aug 04 18:16:49.028818 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:49.028425 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" event={"ID":"7c35d25d-7da3-4ba5-bb5e-937c97e525cf","Type":"ContainerDied","Data":"11dd5349a2f690f9aedd9466190b0b34a79eade57d6162a32468b3175456583c"} Aug 04 18:16:49.028818 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:49.028478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" event={"ID":"7c35d25d-7da3-4ba5-bb5e-937c97e525cf","Type":"ContainerStarted","Data":"f4b90d98805cd1c899499af62bfab03bd2d555580ec9082ce9032e2caaf6e1db"} Aug 04 18:16:50.036312 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:50.035504 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerID="09a290ae61281178255caa29d0cd2c603684904cf76a5209772074aca4b227d6" exitCode=0 Aug 04 18:16:50.036312 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:50.035621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" event={"ID":"7c35d25d-7da3-4ba5-bb5e-937c97e525cf","Type":"ContainerDied","Data":"09a290ae61281178255caa29d0cd2c603684904cf76a5209772074aca4b227d6"} Aug 04 18:16:51.041777 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:51.041741 2575 generic.go:356] "Generic (PLEG): container finished" podID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerID="6e59f84d7bc91180912440aace0148bb62a1cceae10100af32792f866b832a41" exitCode=0 Aug 04 18:16:51.042169 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:51.041788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" event={"ID":"7c35d25d-7da3-4ba5-bb5e-937c97e525cf","Type":"ContainerDied","Data":"6e59f84d7bc91180912440aace0148bb62a1cceae10100af32792f866b832a41"} Aug 04 18:16:52.175001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.174975 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:16:52.307700 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.307634 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-util\") pod \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " Aug 04 18:16:52.307700 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.307691 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rdjh\" (UniqueName: \"kubernetes.io/projected/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-kube-api-access-9rdjh\") pod \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " Aug 04 18:16:52.307904 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.307719 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-bundle\") pod \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\" (UID: \"7c35d25d-7da3-4ba5-bb5e-937c97e525cf\") " Aug 04 18:16:52.308497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.308469 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-bundle" (OuterVolumeSpecName: "bundle") pod "7c35d25d-7da3-4ba5-bb5e-937c97e525cf" (UID: "7c35d25d-7da3-4ba5-bb5e-937c97e525cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:16:52.309930 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.309907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-kube-api-access-9rdjh" (OuterVolumeSpecName: "kube-api-access-9rdjh") pod "7c35d25d-7da3-4ba5-bb5e-937c97e525cf" (UID: "7c35d25d-7da3-4ba5-bb5e-937c97e525cf"). InnerVolumeSpecName "kube-api-access-9rdjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:16:52.313076 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.313054 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-util" (OuterVolumeSpecName: "util") pod "7c35d25d-7da3-4ba5-bb5e-937c97e525cf" (UID: "7c35d25d-7da3-4ba5-bb5e-937c97e525cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:16:52.408855 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.408831 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:52.408855 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.408855 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdjh\" (UniqueName: \"kubernetes.io/projected/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-kube-api-access-9rdjh\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:52.409000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:52.408865 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c35d25d-7da3-4ba5-bb5e-937c97e525cf-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:16:53.053569 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:53.053537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" event={"ID":"7c35d25d-7da3-4ba5-bb5e-937c97e525cf","Type":"ContainerDied","Data":"f4b90d98805cd1c899499af62bfab03bd2d555580ec9082ce9032e2caaf6e1db"} Aug 04 18:16:53.053734 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:53.053582 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b90d98805cd1c899499af62bfab03bd2d555580ec9082ce9032e2caaf6e1db" Aug 04 18:16:53.053734 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:16:53.053583 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw5ll7" Aug 04 18:17:00.027913 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.027876 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf"] Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028167 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="pull" Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028178 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="pull" Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028191 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="util" Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028196 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="util" Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028205 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="extract" Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028210 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="extract" Aug 04 18:17:00.028278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.028266 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c35d25d-7da3-4ba5-bb5e-937c97e525cf" containerName="extract" Aug 04 18:17:00.030332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.030316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.033914 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.033890 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 18:17:00.034044 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.033979 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 18:17:00.034851 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.034837 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 18:17:00.034922 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.034909 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xpz5v\"" Aug 04 18:17:00.035757 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.035738 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 18:17:00.035861 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.035796 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 18:17:00.042691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.042670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf"] Aug 04 18:17:00.073480 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.073450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-cert\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.073614 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.073497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-metrics-cert\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.073614 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.073535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqpf\" (UniqueName: \"kubernetes.io/projected/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-kube-api-access-wsqpf\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.073691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.073648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-manager-config\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.174850 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.174812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-cert\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.175035 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.174861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-metrics-cert\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.175035 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.174919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqpf\" (UniqueName: \"kubernetes.io/projected/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-kube-api-access-wsqpf\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.175035 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.174994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-manager-config\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.175612 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.175592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-manager-config\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.177508 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.177488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-cert\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.177613 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.177582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-metrics-cert\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.186898 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.186864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqpf\" (UniqueName: \"kubernetes.io/projected/261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e-kube-api-access-wsqpf\") pod \"lws-controller-manager-84559d8974-dn8bf\" (UID: \"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e\") " pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.340467 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.340391 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:00.486311 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:00.485825 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf"] Aug 04 18:17:00.491112 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:17:00.491086 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261dbaaf_da0a_4ba7_8ab0_dfd5bc07733e.slice/crio-05c0bd647a5950b5afd765992ad356330dfde596d8da480115dabaee2f4d0c79 WatchSource:0}: Error finding container 05c0bd647a5950b5afd765992ad356330dfde596d8da480115dabaee2f4d0c79: Status 404 returned error can't find the container with id 05c0bd647a5950b5afd765992ad356330dfde596d8da480115dabaee2f4d0c79 Aug 04 18:17:01.089561 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.089525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" event={"ID":"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e","Type":"ContainerStarted","Data":"05c0bd647a5950b5afd765992ad356330dfde596d8da480115dabaee2f4d0c79"} Aug 04 18:17:01.582388 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.582328 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb"] Aug 04 18:17:01.585535 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.585505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.589223 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.589193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qb6pd\"" Aug 04 18:17:01.589342 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.589201 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 18:17:01.590036 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.590006 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 18:17:01.590406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.590268 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 18:17:01.590406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.590323 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 18:17:01.613905 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.613831 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb"] Aug 04 18:17:01.686673 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.686634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7690ee0-8a42-4266-8825-67a07d42b787-tmp\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.686878 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.686706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7690ee0-8a42-4266-8825-67a07d42b787-webhook-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.686878 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.686729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7690ee0-8a42-4266-8825-67a07d42b787-manifests\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.686878 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.686817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5jw\" (UniqueName: \"kubernetes.io/projected/e7690ee0-8a42-4266-8825-67a07d42b787-kube-api-access-2s5jw\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.686878 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.686837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7690ee0-8a42-4266-8825-67a07d42b787-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.787558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.787516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7690ee0-8a42-4266-8825-67a07d42b787-tmp\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.787747 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.787574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7690ee0-8a42-4266-8825-67a07d42b787-webhook-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.787747 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.787690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7690ee0-8a42-4266-8825-67a07d42b787-manifests\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.787888 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.787768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5jw\" (UniqueName: \"kubernetes.io/projected/e7690ee0-8a42-4266-8825-67a07d42b787-kube-api-access-2s5jw\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.787888 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.787789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7690ee0-8a42-4266-8825-67a07d42b787-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.788083 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.788057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7690ee0-8a42-4266-8825-67a07d42b787-manifests\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.789947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.789920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7690ee0-8a42-4266-8825-67a07d42b787-tmp\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.790299 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.790242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7690ee0-8a42-4266-8825-67a07d42b787-webhook-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.790434 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.790305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7690ee0-8a42-4266-8825-67a07d42b787-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.799043 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.798996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5jw\" (UniqueName: \"kubernetes.io/projected/e7690ee0-8a42-4266-8825-67a07d42b787-kube-api-access-2s5jw\") pod \"opendatahub-operator-controller-manager-76bbc69bc6-ls9rb\" (UID: \"e7690ee0-8a42-4266-8825-67a07d42b787\") " pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:01.900217 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:01.900130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:02.256771 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:02.256745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb"] Aug 04 18:17:02.258697 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:17:02.258671 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7690ee0_8a42_4266_8825_67a07d42b787.slice/crio-462d9cb91faa96f1aedce89653be7562b419826c7b197052cf98ee622d5e12a4 WatchSource:0}: Error finding container 462d9cb91faa96f1aedce89653be7562b419826c7b197052cf98ee622d5e12a4: Status 404 returned error can't find the container with id 462d9cb91faa96f1aedce89653be7562b419826c7b197052cf98ee622d5e12a4 Aug 04 18:17:03.106696 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:03.106654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" event={"ID":"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e","Type":"ContainerStarted","Data":"e1a5b5611a820f036a255a34d9789e5b9458acc02b4b26309ba14441b55cb4ea"} Aug 04 18:17:03.106885 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:03.106724 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:03.108313 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:03.108252 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" event={"ID":"e7690ee0-8a42-4266-8825-67a07d42b787","Type":"ContainerStarted","Data":"462d9cb91faa96f1aedce89653be7562b419826c7b197052cf98ee622d5e12a4"} Aug 04 18:17:03.129109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:03.129053 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" podStartSLOduration=1.472045393 podStartE2EDuration="3.12903365s" totalImagesPullingTime="1.656988257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:03.127548302 +0000 UTC m=+388.274364563" watchObservedRunningTime="2026-08-04 18:17:03.12903365 +0000 UTC m=+388.275849957" Aug 04 18:17:06.127458 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:06.127418 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" event={"ID":"e7690ee0-8a42-4266-8825-67a07d42b787","Type":"ContainerStarted","Data":"f909b837e99f43f00e12de8213684af6efdbef1d62732e7f75776f9b8ac1422d"} Aug 04 18:17:07.133271 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:07.133236 2575 generic.go:356] "Generic (PLEG): container finished" podID="e7690ee0-8a42-4266-8825-67a07d42b787" containerID="f909b837e99f43f00e12de8213684af6efdbef1d62732e7f75776f9b8ac1422d" exitCode=0 Aug 04 18:17:07.133675 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:07.133290 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" event={"ID":"e7690ee0-8a42-4266-8825-67a07d42b787","Type":"ContainerDied","Data":"f909b837e99f43f00e12de8213684af6efdbef1d62732e7f75776f9b8ac1422d"} Aug 04 18:17:08.139597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:08.139516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" event={"ID":"e7690ee0-8a42-4266-8825-67a07d42b787","Type":"ContainerStarted","Data":"4d4d099aed613ba0e642e7d5ab0bc29050316fc34cd684693e439ef2ffe45095"} Aug 04 18:17:08.139942 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:08.139695 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:08.167539 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:08.167490 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" podStartSLOduration=2.539463555 podStartE2EDuration="7.167474394s" totalImagesPullingTime="3.6075097s" totalInitContainerRuntime="1.020501139s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:08.166037828 +0000 UTC m=+393.312854090" watchObservedRunningTime="2026-08-04 18:17:08.167474394 +0000 UTC m=+393.314290654" Aug 04 18:17:14.123754 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:14.123708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:17:19.146772 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:19.146742 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76bbc69bc6-ls9rb" Aug 04 18:17:31.973806 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.973772 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5"] Aug 04 18:17:31.977113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.977096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:31.986390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.986349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 18:17:31.986531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.986397 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 18:17:31.986531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.986349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 18:17:31.986850 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.986833 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 18:17:31.987089 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.987072 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tmws8\"" Aug 04 18:17:31.994957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:31.994932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5"] Aug 04 18:17:32.061268 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.061268 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d16691-b496-43ea-960a-afac7fecc344-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.061492 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggx6s\" (UniqueName: \"kubernetes.io/projected/76d16691-b496-43ea-960a-afac7fecc344-kube-api-access-ggx6s\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.061492 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d16691-b496-43ea-960a-afac7fecc344-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.061565 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.061565 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.061565 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.061541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d16691-b496-43ea-960a-afac7fecc344-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.162755 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.162716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.162946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.162760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.162946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.162891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d16691-b496-43ea-960a-afac7fecc344-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.163053 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.162956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.163053 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.162984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d16691-b496-43ea-960a-afac7fecc344-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.163053 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.163000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggx6s\" (UniqueName: \"kubernetes.io/projected/76d16691-b496-43ea-960a-afac7fecc344-kube-api-access-ggx6s\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.163053 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.163022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d16691-b496-43ea-960a-afac7fecc344-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.163691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.163617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d16691-b496-43ea-960a-afac7fecc344-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.165433 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.165411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.165531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.165458 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.165531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.165522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d16691-b496-43ea-960a-afac7fecc344-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.165621 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.165569 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d16691-b496-43ea-960a-afac7fecc344-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.173720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.173686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggx6s\" (UniqueName: \"kubernetes.io/projected/76d16691-b496-43ea-960a-afac7fecc344-kube-api-access-ggx6s\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.176328 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.176308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d16691-b496-43ea-960a-afac7fecc344-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b9fc5\" (UID: \"76d16691-b496-43ea-960a-afac7fecc344\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.285807 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.285721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:32.437105 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:32.437078 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5"] Aug 04 18:17:32.438916 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:17:32.438883 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d16691_b496_43ea_960a_afac7fecc344.slice/crio-c30329b5c230f12f98c6f9a89ec30ccfc7dc5b17ef5072486e76ef75fbcfe8b5 WatchSource:0}: Error finding container c30329b5c230f12f98c6f9a89ec30ccfc7dc5b17ef5072486e76ef75fbcfe8b5: Status 404 returned error can't find the container with id c30329b5c230f12f98c6f9a89ec30ccfc7dc5b17ef5072486e76ef75fbcfe8b5 Aug 04 18:17:33.256642 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:33.256605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" event={"ID":"76d16691-b496-43ea-960a-afac7fecc344","Type":"ContainerStarted","Data":"c30329b5c230f12f98c6f9a89ec30ccfc7dc5b17ef5072486e76ef75fbcfe8b5"} Aug 04 18:17:35.207837 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:35.207799 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:17:35.208124 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:35.207868 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:17:36.275067 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:36.275029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" event={"ID":"76d16691-b496-43ea-960a-afac7fecc344","Type":"ContainerStarted","Data":"5d3a3c6bdf6cb69f511ad75bbbf36a271c7425da5b602a8a0207c75d3f391134"} Aug 04 18:17:36.275555 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:36.275253 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:17:36.276907 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:36.276879 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-b9fc5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:17:36.277013 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:36.276934 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" podUID="76d16691-b496-43ea-960a-afac7fecc344" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:17:36.307849 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:36.307803 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" podStartSLOduration=2.541246734 podStartE2EDuration="5.307788033s" totalImagesPullingTime="2.766541299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:36.305889311 +0000 UTC m=+421.452705571" watchObservedRunningTime="2026-08-04 18:17:36.307788033 +0000 UTC m=+421.454604293" Aug 04 18:17:37.281355 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:17:37.281323 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b9fc5" Aug 04 18:18:27.795386 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.795331 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4"] Aug 04 18:18:27.798693 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.798677 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:27.801440 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.801414 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 18:18:27.801580 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.801419 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6bz9z\"" Aug 04 18:18:27.801580 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.801419 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 18:18:27.808014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.807986 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4"] Aug 04 18:18:27.897783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.897751 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74"] Aug 04 18:18:27.901290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.901274 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:27.911037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.911012 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74"] Aug 04 18:18:27.930189 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.930162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:27.930294 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.930193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:27.930335 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.930287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfbt2\" (UniqueName: \"kubernetes.io/projected/b2f3f784-93a5-4c18-979e-c6c5439d7d11-kube-api-access-xfbt2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:27.997676 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:27.997647 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv"] Aug 04 18:18:28.001086 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.001070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.010314 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.010293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv"] Aug 04 18:18:28.031215 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfbt2\" (UniqueName: \"kubernetes.io/projected/b2f3f784-93a5-4c18-979e-c6c5439d7d11-kube-api-access-xfbt2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.031333 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.031333 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.031333 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.031490 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.031490 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfgb\" (UniqueName: \"kubernetes.io/projected/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-kube-api-access-swfgb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.031638 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.031676 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.031662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.042243 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.042225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfbt2\" (UniqueName: \"kubernetes.io/projected/b2f3f784-93a5-4c18-979e-c6c5439d7d11-kube-api-access-xfbt2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.108813 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.108743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:28.109940 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.109910 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5"] Aug 04 18:18:28.113252 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.113235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.129553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.128142 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5"] Aug 04 18:18:28.134217 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfgb\" (UniqueName: \"kubernetes.io/projected/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-kube-api-access-swfgb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.135332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc29n\" (UniqueName: \"kubernetes.io/projected/1ec5e765-66af-49dd-8753-9f811b3464e5-kube-api-access-jc29n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.135332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.135332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.135332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.135332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.135332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.134979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.135715 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.135626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.148214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.148183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfgb\" (UniqueName: \"kubernetes.io/projected/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-kube-api-access-swfgb\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.211684 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.211646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:28.235217 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.235356 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc29n\" (UniqueName: \"kubernetes.io/projected/1ec5e765-66af-49dd-8753-9f811b3464e5-kube-api-access-jc29n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.235356 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.235497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.235549 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.235612 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g64p8\" (UniqueName: \"kubernetes.io/projected/e66cb0bf-3cf9-4db3-91d1-891ef882407b-kube-api-access-g64p8\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.235976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.235951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.236096 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.236020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.248640 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.248611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc29n\" (UniqueName: \"kubernetes.io/projected/1ec5e765-66af-49dd-8753-9f811b3464e5-kube-api-access-jc29n\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.264046 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.264017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4"] Aug 04 18:18:28.268183 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:18:28.268137 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f3f784_93a5_4c18_979e_c6c5439d7d11.slice/crio-fefdb191f3a2768dcb510a20e37b56e903c5fbb70d392839055f5977ae013a21 WatchSource:0}: Error finding container fefdb191f3a2768dcb510a20e37b56e903c5fbb70d392839055f5977ae013a21: Status 404 returned error can't find the container with id fefdb191f3a2768dcb510a20e37b56e903c5fbb70d392839055f5977ae013a21 Aug 04 18:18:28.310465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.310428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:28.336836 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.336639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g64p8\" (UniqueName: \"kubernetes.io/projected/e66cb0bf-3cf9-4db3-91d1-891ef882407b-kube-api-access-g64p8\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.336836 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.336704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.336836 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.336741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.337179 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.337156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.337251 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.337218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.346718 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.346691 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g64p8\" (UniqueName: \"kubernetes.io/projected/e66cb0bf-3cf9-4db3-91d1-891ef882407b-kube-api-access-g64p8\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.371329 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.370847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74"] Aug 04 18:18:28.375571 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:18:28.375468 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdae031a_a125_40e8_b4ec_8c51cd9e0a3d.slice/crio-cdf9316a9f8856221be7a2380c7a2dadbff1e6af6f1625a721dbefaaba9fbc03 WatchSource:0}: Error finding container cdf9316a9f8856221be7a2380c7a2dadbff1e6af6f1625a721dbefaaba9fbc03: Status 404 returned error can't find the container with id cdf9316a9f8856221be7a2380c7a2dadbff1e6af6f1625a721dbefaaba9fbc03 Aug 04 18:18:28.447552 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.447523 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:28.470275 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.470248 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv"] Aug 04 18:18:28.470944 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:18:28.470913 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec5e765_66af_49dd_8753_9f811b3464e5.slice/crio-729735e998c6e133eef1f53c0f9ed540d0d7733cab261b97e66ac0f16d082536 WatchSource:0}: Error finding container 729735e998c6e133eef1f53c0f9ed540d0d7733cab261b97e66ac0f16d082536: Status 404 returned error can't find the container with id 729735e998c6e133eef1f53c0f9ed540d0d7733cab261b97e66ac0f16d082536 Aug 04 18:18:28.517605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.517570 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" event={"ID":"1ec5e765-66af-49dd-8753-9f811b3464e5","Type":"ContainerStarted","Data":"729735e998c6e133eef1f53c0f9ed540d0d7733cab261b97e66ac0f16d082536"} Aug 04 18:18:28.519282 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.519246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" event={"ID":"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d","Type":"ContainerStarted","Data":"e9536bbace6c6e84e299cf0f9251de0079c167a8fd2ca721af20bc520f9eef9b"} Aug 04 18:18:28.519437 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.519306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" event={"ID":"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d","Type":"ContainerStarted","Data":"cdf9316a9f8856221be7a2380c7a2dadbff1e6af6f1625a721dbefaaba9fbc03"} Aug 04 18:18:28.520921 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.520895 2575 generic.go:356] "Generic (PLEG): container finished" podID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerID="f7cca1dfa5957cd306b35605440f5a6308836cb4643e93690778aea07d33e7cd" exitCode=0 Aug 04 18:18:28.521001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.520941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" event={"ID":"b2f3f784-93a5-4c18-979e-c6c5439d7d11","Type":"ContainerDied","Data":"f7cca1dfa5957cd306b35605440f5a6308836cb4643e93690778aea07d33e7cd"} Aug 04 18:18:28.521001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.520981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" event={"ID":"b2f3f784-93a5-4c18-979e-c6c5439d7d11","Type":"ContainerStarted","Data":"fefdb191f3a2768dcb510a20e37b56e903c5fbb70d392839055f5977ae013a21"} Aug 04 18:18:28.599108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:28.599084 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5"] Aug 04 18:18:28.646420 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:18:28.646328 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66cb0bf_3cf9_4db3_91d1_891ef882407b.slice/crio-e3920ecd0a7c2e3dab0da40778f7c4bbc0633d5e656de7c1d58b6357e7da6435 WatchSource:0}: Error finding container e3920ecd0a7c2e3dab0da40778f7c4bbc0633d5e656de7c1d58b6357e7da6435: Status 404 returned error can't find the container with id e3920ecd0a7c2e3dab0da40778f7c4bbc0633d5e656de7c1d58b6357e7da6435 Aug 04 18:18:29.528166 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.528129 2575 generic.go:356] "Generic (PLEG): container finished" podID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerID="e9536bbace6c6e84e299cf0f9251de0079c167a8fd2ca721af20bc520f9eef9b" exitCode=0 Aug 04 18:18:29.528597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.528211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" event={"ID":"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d","Type":"ContainerDied","Data":"e9536bbace6c6e84e299cf0f9251de0079c167a8fd2ca721af20bc520f9eef9b"} Aug 04 18:18:29.529616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.529595 2575 generic.go:356] "Generic (PLEG): container finished" podID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerID="e592bb259a5c2ae1954eb6d0be19f224baeeb05e3d61cda2d7a15c686e9ee80d" exitCode=0 Aug 04 18:18:29.529701 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.529675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" event={"ID":"1ec5e765-66af-49dd-8753-9f811b3464e5","Type":"ContainerDied","Data":"e592bb259a5c2ae1954eb6d0be19f224baeeb05e3d61cda2d7a15c686e9ee80d"} Aug 04 18:18:29.531290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.531223 2575 generic.go:356] "Generic (PLEG): container finished" podID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerID="cd21cb6a68404a61cf44f066de80d49e4cb56baf38b4bcb21d16f0397f7c269a" exitCode=0 Aug 04 18:18:29.531290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.531260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" event={"ID":"e66cb0bf-3cf9-4db3-91d1-891ef882407b","Type":"ContainerDied","Data":"cd21cb6a68404a61cf44f066de80d49e4cb56baf38b4bcb21d16f0397f7c269a"} Aug 04 18:18:29.531390 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:29.531319 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" event={"ID":"e66cb0bf-3cf9-4db3-91d1-891ef882407b","Type":"ContainerStarted","Data":"e3920ecd0a7c2e3dab0da40778f7c4bbc0633d5e656de7c1d58b6357e7da6435"} Aug 04 18:18:30.538529 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:30.538497 2575 generic.go:356] "Generic (PLEG): container finished" podID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerID="d0ab47c403c0bc7be5a44bcbd6929c422a2b3de9a3544d942eec1c15cf1a3273" exitCode=0 Aug 04 18:18:30.538893 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:30.538580 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" event={"ID":"1ec5e765-66af-49dd-8753-9f811b3464e5","Type":"ContainerDied","Data":"d0ab47c403c0bc7be5a44bcbd6929c422a2b3de9a3544d942eec1c15cf1a3273"} Aug 04 18:18:30.540598 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:30.540578 2575 generic.go:356] "Generic (PLEG): container finished" podID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerID="4e14261c23eb78c68eb2fa0ce67a16564eb179d9f998bc09c0785e385b67f58d" exitCode=0 Aug 04 18:18:30.540733 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:30.540651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" event={"ID":"e66cb0bf-3cf9-4db3-91d1-891ef882407b","Type":"ContainerDied","Data":"4e14261c23eb78c68eb2fa0ce67a16564eb179d9f998bc09c0785e385b67f58d"} Aug 04 18:18:30.542633 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:30.542614 2575 generic.go:356] "Generic (PLEG): container finished" podID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerID="efd0bf3a9a076829c4dcd1e7adfc74b9bfc32a0739d9c67f10ee157e9a60b6ca" exitCode=0 Aug 04 18:18:30.542736 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:30.542641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" event={"ID":"b2f3f784-93a5-4c18-979e-c6c5439d7d11","Type":"ContainerDied","Data":"efd0bf3a9a076829c4dcd1e7adfc74b9bfc32a0739d9c67f10ee157e9a60b6ca"} Aug 04 18:18:31.549600 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.549563 2575 generic.go:356] "Generic (PLEG): container finished" podID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerID="66baa8b7f9777078be06b379d05203ff2ed25820504e3eeb69695e8b0001d2d1" exitCode=0 Aug 04 18:18:31.550016 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.549650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" event={"ID":"1ec5e765-66af-49dd-8753-9f811b3464e5","Type":"ContainerDied","Data":"66baa8b7f9777078be06b379d05203ff2ed25820504e3eeb69695e8b0001d2d1"} Aug 04 18:18:31.551620 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.551596 2575 generic.go:356] "Generic (PLEG): container finished" podID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerID="552814537ce547bd8ab75b0dcf9ca9f4dd580b5301aaf343bb0f366320dddcfb" exitCode=0 Aug 04 18:18:31.551815 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.551674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" event={"ID":"e66cb0bf-3cf9-4db3-91d1-891ef882407b","Type":"ContainerDied","Data":"552814537ce547bd8ab75b0dcf9ca9f4dd580b5301aaf343bb0f366320dddcfb"} Aug 04 18:18:31.553452 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.553422 2575 generic.go:356] "Generic (PLEG): container finished" podID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerID="a7fc1b3d9c4483fc7ce430b2aa34493caf7fc0fa5358d29505caca063e9498be" exitCode=0 Aug 04 18:18:31.553556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.553488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" event={"ID":"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d","Type":"ContainerDied","Data":"a7fc1b3d9c4483fc7ce430b2aa34493caf7fc0fa5358d29505caca063e9498be"} Aug 04 18:18:31.555750 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.555730 2575 generic.go:356] "Generic (PLEG): container finished" podID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerID="5233d251986f0182f3e73f1fb998634853233a76811ebde99812a234a385cfa0" exitCode=0 Aug 04 18:18:31.555850 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:31.555764 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" event={"ID":"b2f3f784-93a5-4c18-979e-c6c5439d7d11","Type":"ContainerDied","Data":"5233d251986f0182f3e73f1fb998634853233a76811ebde99812a234a385cfa0"} Aug 04 18:18:32.564044 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.564011 2575 generic.go:356] "Generic (PLEG): container finished" podID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerID="76e4eb61b1f1af52ceaf8108f93ff67a4f6ae90a4eb79bdc453bccc58d2cc85c" exitCode=0 Aug 04 18:18:32.564540 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.564099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" event={"ID":"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d","Type":"ContainerDied","Data":"76e4eb61b1f1af52ceaf8108f93ff67a4f6ae90a4eb79bdc453bccc58d2cc85c"} Aug 04 18:18:32.709046 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.709018 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:32.755570 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.755547 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:32.758536 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.758518 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:32.780824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.780796 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g64p8\" (UniqueName: \"kubernetes.io/projected/e66cb0bf-3cf9-4db3-91d1-891ef882407b-kube-api-access-g64p8\") pod \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " Aug 04 18:18:32.780959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.780888 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-util\") pod \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " Aug 04 18:18:32.780959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.780913 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-bundle\") pod \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\" (UID: \"e66cb0bf-3cf9-4db3-91d1-891ef882407b\") " Aug 04 18:18:32.781655 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.781623 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-bundle" (OuterVolumeSpecName: "bundle") pod "e66cb0bf-3cf9-4db3-91d1-891ef882407b" (UID: "e66cb0bf-3cf9-4db3-91d1-891ef882407b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:32.783298 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.783273 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66cb0bf-3cf9-4db3-91d1-891ef882407b-kube-api-access-g64p8" (OuterVolumeSpecName: "kube-api-access-g64p8") pod "e66cb0bf-3cf9-4db3-91d1-891ef882407b" (UID: "e66cb0bf-3cf9-4db3-91d1-891ef882407b"). InnerVolumeSpecName "kube-api-access-g64p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:18:32.786755 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.786726 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-util" (OuterVolumeSpecName: "util") pod "e66cb0bf-3cf9-4db3-91d1-891ef882407b" (UID: "e66cb0bf-3cf9-4db3-91d1-891ef882407b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:32.882532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882442 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-bundle\") pod \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " Aug 04 18:18:32.882532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882500 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfbt2\" (UniqueName: \"kubernetes.io/projected/b2f3f784-93a5-4c18-979e-c6c5439d7d11-kube-api-access-xfbt2\") pod \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " Aug 04 18:18:32.882532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882531 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc29n\" (UniqueName: \"kubernetes.io/projected/1ec5e765-66af-49dd-8753-9f811b3464e5-kube-api-access-jc29n\") pod \"1ec5e765-66af-49dd-8753-9f811b3464e5\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " Aug 04 18:18:32.882815 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882564 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-util\") pod \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\" (UID: \"b2f3f784-93a5-4c18-979e-c6c5439d7d11\") " Aug 04 18:18:32.882815 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-bundle\") pod \"1ec5e765-66af-49dd-8753-9f811b3464e5\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " Aug 04 18:18:32.882815 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882660 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-util\") pod \"1ec5e765-66af-49dd-8753-9f811b3464e5\" (UID: \"1ec5e765-66af-49dd-8753-9f811b3464e5\") " Aug 04 18:18:32.882957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882881 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g64p8\" (UniqueName: \"kubernetes.io/projected/e66cb0bf-3cf9-4db3-91d1-891ef882407b-kube-api-access-g64p8\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.882957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882904 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.882957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.882919 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e66cb0bf-3cf9-4db3-91d1-891ef882407b-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.883155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.883119 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-bundle" (OuterVolumeSpecName: "bundle") pod "b2f3f784-93a5-4c18-979e-c6c5439d7d11" (UID: "b2f3f784-93a5-4c18-979e-c6c5439d7d11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:32.883155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.883137 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-bundle" (OuterVolumeSpecName: "bundle") pod "1ec5e765-66af-49dd-8753-9f811b3464e5" (UID: "1ec5e765-66af-49dd-8753-9f811b3464e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:32.884923 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.884896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec5e765-66af-49dd-8753-9f811b3464e5-kube-api-access-jc29n" (OuterVolumeSpecName: "kube-api-access-jc29n") pod "1ec5e765-66af-49dd-8753-9f811b3464e5" (UID: "1ec5e765-66af-49dd-8753-9f811b3464e5"). InnerVolumeSpecName "kube-api-access-jc29n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:18:32.885019 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.884965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f3f784-93a5-4c18-979e-c6c5439d7d11-kube-api-access-xfbt2" (OuterVolumeSpecName: "kube-api-access-xfbt2") pod "b2f3f784-93a5-4c18-979e-c6c5439d7d11" (UID: "b2f3f784-93a5-4c18-979e-c6c5439d7d11"). InnerVolumeSpecName "kube-api-access-xfbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:18:32.887743 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.887722 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-util" (OuterVolumeSpecName: "util") pod "1ec5e765-66af-49dd-8753-9f811b3464e5" (UID: "1ec5e765-66af-49dd-8753-9f811b3464e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:32.888411 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.888391 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-util" (OuterVolumeSpecName: "util") pod "b2f3f784-93a5-4c18-979e-c6c5439d7d11" (UID: "b2f3f784-93a5-4c18-979e-c6c5439d7d11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:32.983623 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.983590 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc29n\" (UniqueName: \"kubernetes.io/projected/1ec5e765-66af-49dd-8753-9f811b3464e5-kube-api-access-jc29n\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.983623 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.983617 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.983623 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.983627 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.983623 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.983634 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec5e765-66af-49dd-8753-9f811b3464e5-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.983882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.983643 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f3f784-93a5-4c18-979e-c6c5439d7d11-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:32.983882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:32.983652 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfbt2\" (UniqueName: \"kubernetes.io/projected/b2f3f784-93a5-4c18-979e-c6c5439d7d11-kube-api-access-xfbt2\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:33.570808 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.570776 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" Aug 04 18:18:33.570808 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.570800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbkjjq4" event={"ID":"b2f3f784-93a5-4c18-979e-c6c5439d7d11","Type":"ContainerDied","Data":"fefdb191f3a2768dcb510a20e37b56e903c5fbb70d392839055f5977ae013a21"} Aug 04 18:18:33.571308 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.570850 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fefdb191f3a2768dcb510a20e37b56e903c5fbb70d392839055f5977ae013a21" Aug 04 18:18:33.573105 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.573075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" event={"ID":"1ec5e765-66af-49dd-8753-9f811b3464e5","Type":"ContainerDied","Data":"729735e998c6e133eef1f53c0f9ed540d0d7733cab261b97e66ac0f16d082536"} Aug 04 18:18:33.573245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.573122 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="729735e998c6e133eef1f53c0f9ed540d0d7733cab261b97e66ac0f16d082536" Aug 04 18:18:33.573245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.573138 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a07p6mv" Aug 04 18:18:33.575250 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.575233 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" Aug 04 18:18:33.575377 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.575272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053f8kl5" event={"ID":"e66cb0bf-3cf9-4db3-91d1-891ef882407b","Type":"ContainerDied","Data":"e3920ecd0a7c2e3dab0da40778f7c4bbc0633d5e656de7c1d58b6357e7da6435"} Aug 04 18:18:33.575377 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.575314 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3920ecd0a7c2e3dab0da40778f7c4bbc0633d5e656de7c1d58b6357e7da6435" Aug 04 18:18:33.711451 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.711425 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:33.789529 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.789496 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-bundle\") pod \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " Aug 04 18:18:33.789717 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.789563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-util\") pod \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " Aug 04 18:18:33.789717 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.789582 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swfgb\" (UniqueName: \"kubernetes.io/projected/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-kube-api-access-swfgb\") pod \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\" (UID: \"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d\") " Aug 04 18:18:33.790183 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.790142 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-bundle" (OuterVolumeSpecName: "bundle") pod "bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" (UID: "bdae031a-a125-40e8-b4ec-8c51cd9e0a3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:33.792014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.791987 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-kube-api-access-swfgb" (OuterVolumeSpecName: "kube-api-access-swfgb") pod "bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" (UID: "bdae031a-a125-40e8-b4ec-8c51cd9e0a3d"). InnerVolumeSpecName "kube-api-access-swfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:18:33.794376 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.794337 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-util" (OuterVolumeSpecName: "util") pod "bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" (UID: "bdae031a-a125-40e8-b4ec-8c51cd9e0a3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:33.891004 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.890912 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-util\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:33.891004 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.890942 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfgb\" (UniqueName: \"kubernetes.io/projected/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-kube-api-access-swfgb\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:33.891004 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:33.890953 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdae031a-a125-40e8-b4ec-8c51cd9e0a3d-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:18:34.581975 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:34.581947 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" Aug 04 18:18:34.581975 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:34.581969 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cxkw74" event={"ID":"bdae031a-a125-40e8-b4ec-8c51cd9e0a3d","Type":"ContainerDied","Data":"cdf9316a9f8856221be7a2380c7a2dadbff1e6af6f1625a721dbefaaba9fbc03"} Aug 04 18:18:34.582424 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:18:34.582020 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf9316a9f8856221be7a2380c7a2dadbff1e6af6f1625a721dbefaaba9fbc03" Aug 04 18:19:05.922580 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922544 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2"] Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922847 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922858 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922869 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922874 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922884 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922889 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922896 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922903 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922913 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922919 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922930 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922935 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922941 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922955 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922960 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922967 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922972 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922978 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922984 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922991 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.922996 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="pull" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.923002 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.923008 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="util" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.923057 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66cb0bf-3cf9-4db3-91d1-891ef882407b" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.923066 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f3f784-93a5-4c18-979e-c6c5439d7d11" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.923073 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec5e765-66af-49dd-8753-9f811b3464e5" containerName="extract" Aug 04 18:19:05.923075 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.923080 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdae031a-a125-40e8-b4ec-8c51cd9e0a3d" containerName="extract" Aug 04 18:19:05.926042 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.926014 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:05.928765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.928745 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 18:19:05.928863 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.928838 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rrnp5\"" Aug 04 18:19:05.928916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.928875 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 18:19:05.954450 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.954427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2"] Aug 04 18:19:05.993204 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.993170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ba55a57-860e-4efc-b0c1-173f462a25a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-vpfx2\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:05.993340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:05.993239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwdn\" (UniqueName: \"kubernetes.io/projected/6ba55a57-860e-4efc-b0c1-173f462a25a4-kube-api-access-wvwdn\") pod \"kuadrant-operator-controller-manager-68cf589589-vpfx2\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:06.094059 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.094025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwdn\" (UniqueName: \"kubernetes.io/projected/6ba55a57-860e-4efc-b0c1-173f462a25a4-kube-api-access-wvwdn\") pod \"kuadrant-operator-controller-manager-68cf589589-vpfx2\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:06.094264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.094099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ba55a57-860e-4efc-b0c1-173f462a25a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-vpfx2\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:06.094513 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.094490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ba55a57-860e-4efc-b0c1-173f462a25a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-vpfx2\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:06.109926 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.109897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwdn\" (UniqueName: \"kubernetes.io/projected/6ba55a57-860e-4efc-b0c1-173f462a25a4-kube-api-access-wvwdn\") pod \"kuadrant-operator-controller-manager-68cf589589-vpfx2\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:06.236424 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.236332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:06.393941 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.393916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2"] Aug 04 18:19:06.396933 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:19:06.396905 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba55a57_860e_4efc_b0c1_173f462a25a4.slice/crio-8e6a66e1316af4edb8ae8ac3908324aa1251545bd80fe8fe8d918b64c6fed676 WatchSource:0}: Error finding container 8e6a66e1316af4edb8ae8ac3908324aa1251545bd80fe8fe8d918b64c6fed676: Status 404 returned error can't find the container with id 8e6a66e1316af4edb8ae8ac3908324aa1251545bd80fe8fe8d918b64c6fed676 Aug 04 18:19:06.747215 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:06.747183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" event={"ID":"6ba55a57-860e-4efc-b0c1-173f462a25a4","Type":"ContainerStarted","Data":"8e6a66e1316af4edb8ae8ac3908324aa1251545bd80fe8fe8d918b64c6fed676"} Aug 04 18:19:10.776295 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:10.776262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" event={"ID":"6ba55a57-860e-4efc-b0c1-173f462a25a4","Type":"ContainerStarted","Data":"495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73"} Aug 04 18:19:10.776755 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:10.776351 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:19:10.799254 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:10.799202 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" podStartSLOduration=1.873798549 podStartE2EDuration="5.799186688s" totalImagesPullingTime="3.925388139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:10.797182163 +0000 UTC m=+515.943998423" watchObservedRunningTime="2026-08-04 18:19:10.799186688 +0000 UTC m=+515.946003039" Aug 04 18:19:21.783479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:19:21.783450 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:20:35.292959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:20:35.292914 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:20:35.294137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:20:35.294114 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:21:20.617966 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.617931 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq"] Aug 04 18:21:20.621667 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.621645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.624438 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.624413 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 18:21:20.624576 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.624413 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 04 18:21:20.625392 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.625340 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 18:21:20.625506 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.625397 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6vn8c\"" Aug 04 18:21:20.635822 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.635790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.635962 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.635835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.635962 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.635892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26c7w\" (UniqueName: \"kubernetes.io/projected/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-kube-api-access-26c7w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.636065 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.635998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.636109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.636061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.636155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.636138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.637913 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.637892 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq"] Aug 04 18:21:20.737379 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26c7w\" (UniqueName: \"kubernetes.io/projected/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-kube-api-access-26c7w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737902 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.737959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737934 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.738002 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.737983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.740064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.740030 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.740339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.740315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.747085 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.747057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26c7w\" (UniqueName: \"kubernetes.io/projected/2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8-kube-api-access-26c7w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq\" (UID: \"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:20.932463 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:20.932422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:21.076874 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:21.076845 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq"] Aug 04 18:21:21.078864 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:21:21.078835 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dab5fb1_d3fb_44cb_8cd9_d0bbeffc81e8.slice/crio-f1278ff0fde8d8b7b264254cca47714a74cfbf0011f86dfe2a446101883c5184 WatchSource:0}: Error finding container f1278ff0fde8d8b7b264254cca47714a74cfbf0011f86dfe2a446101883c5184: Status 404 returned error can't find the container with id f1278ff0fde8d8b7b264254cca47714a74cfbf0011f86dfe2a446101883c5184 Aug 04 18:21:21.080469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:21.080448 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:21:21.469235 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:21.469197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" event={"ID":"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8","Type":"ContainerStarted","Data":"f1278ff0fde8d8b7b264254cca47714a74cfbf0011f86dfe2a446101883c5184"} Aug 04 18:21:22.476275 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:22.476190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" event={"ID":"2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8","Type":"ContainerStarted","Data":"88465f37d6ead0cc181e5fe8c72ee0ad54c9cd33f148ea73b52fa8a2546afd31"} Aug 04 18:21:22.496704 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:22.496656 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" podStartSLOduration=1.390712786 podStartE2EDuration="2.496640879s" totalImagesPullingTime="1.105928093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:22.496093676 +0000 UTC m=+647.642909938" watchObservedRunningTime="2026-08-04 18:21:22.496640879 +0000 UTC m=+647.643457139" Aug 04 18:21:30.932883 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:30.932845 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:30.933286 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:30.932907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:30.945143 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:30.945118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:21:31.534677 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:21:31.534649 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq" Aug 04 18:22:00.331406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.331358 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww"] Aug 04 18:22:00.334824 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.334809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.337810 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.337787 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 18:22:00.361930 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.361899 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww"] Aug 04 18:22:00.501992 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.501952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.502223 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.502057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.502223 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.502084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980f0560-6865-4966-8356-225ec2a7fee9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.502223 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.502142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.502416 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.502226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.502416 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.502260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnq5t\" (UniqueName: \"kubernetes.io/projected/980f0560-6865-4966-8356-225ec2a7fee9-kube-api-access-gnq5t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603393 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603393 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980f0560-6865-4966-8356-225ec2a7fee9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq5t\" (UniqueName: \"kubernetes.io/projected/980f0560-6865-4966-8356-225ec2a7fee9-kube-api-access-gnq5t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603834 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603834 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.603945 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.603929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.605848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.605829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/980f0560-6865-4966-8356-225ec2a7fee9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.606001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.605983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/980f0560-6865-4966-8356-225ec2a7fee9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.615538 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.615507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnq5t\" (UniqueName: \"kubernetes.io/projected/980f0560-6865-4966-8356-225ec2a7fee9-kube-api-access-gnq5t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww\" (UID: \"980f0560-6865-4966-8356-225ec2a7fee9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.645212 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.645187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:00.787920 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:00.787895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww"] Aug 04 18:22:00.790579 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:22:00.790553 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980f0560_6865_4966_8356_225ec2a7fee9.slice/crio-d67c3bcdfd0925847b38471a74ebd0b8df31cfc81bb1bc8d5efc8703b3579942 WatchSource:0}: Error finding container d67c3bcdfd0925847b38471a74ebd0b8df31cfc81bb1bc8d5efc8703b3579942: Status 404 returned error can't find the container with id d67c3bcdfd0925847b38471a74ebd0b8df31cfc81bb1bc8d5efc8703b3579942 Aug 04 18:22:01.677336 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:01.677296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" event={"ID":"980f0560-6865-4966-8356-225ec2a7fee9","Type":"ContainerStarted","Data":"d463fdf84d110ff163fbc21a78d1909e02b84ac01595d74e39b2bf73ad782e06"} Aug 04 18:22:01.677717 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:01.677358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" event={"ID":"980f0560-6865-4966-8356-225ec2a7fee9","Type":"ContainerStarted","Data":"d67c3bcdfd0925847b38471a74ebd0b8df31cfc81bb1bc8d5efc8703b3579942"} Aug 04 18:22:01.703101 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:01.703043 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" podStartSLOduration=1.4920165619999999 podStartE2EDuration="1.70302799s" totalImagesPullingTime="211.011428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:01.700266711 +0000 UTC m=+686.847082987" watchObservedRunningTime="2026-08-04 18:22:01.70302799 +0000 UTC m=+686.849844250" Aug 04 18:22:10.646059 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:10.646028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:10.646507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:10.646088 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:10.658411 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:10.658389 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:22:10.734072 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:22:10.734045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww" Aug 04 18:25:35.327342 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:25:35.327268 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:25:35.330033 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:25:35.330009 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:27:29.521511 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.521472 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dfb67b584-mh89d"] Aug 04 18:27:29.524944 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.524927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.555016 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.554984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfb67b584-mh89d"] Aug 04 18:27:29.646279 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b76101e4-4f7b-457f-8877-607fde9266e9-console-serving-cert\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.646470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-oauth-serving-cert\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.646470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-console-config\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.646470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b76101e4-4f7b-457f-8877-607fde9266e9-console-oauth-config\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.646470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-trusted-ca-bundle\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.646605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmh4\" (UniqueName: \"kubernetes.io/projected/b76101e4-4f7b-457f-8877-607fde9266e9-kube-api-access-wlmh4\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.646605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.646544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-service-ca\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747545 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-service-ca\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747545 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b76101e4-4f7b-457f-8877-607fde9266e9-console-serving-cert\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747802 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-oauth-serving-cert\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747802 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-console-config\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747802 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b76101e4-4f7b-457f-8877-607fde9266e9-console-oauth-config\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747973 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-trusted-ca-bundle\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.747973 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.747856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmh4\" (UniqueName: \"kubernetes.io/projected/b76101e4-4f7b-457f-8877-607fde9266e9-kube-api-access-wlmh4\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.748503 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.748432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-service-ca\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.748503 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.748483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-oauth-serving-cert\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.748706 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.748532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-console-config\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.748706 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.748649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b76101e4-4f7b-457f-8877-607fde9266e9-trusted-ca-bundle\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.750317 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.750297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b76101e4-4f7b-457f-8877-607fde9266e9-console-serving-cert\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.750497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.750479 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b76101e4-4f7b-457f-8877-607fde9266e9-console-oauth-config\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.757002 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.756979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmh4\" (UniqueName: \"kubernetes.io/projected/b76101e4-4f7b-457f-8877-607fde9266e9-kube-api-access-wlmh4\") pod \"console-dfb67b584-mh89d\" (UID: \"b76101e4-4f7b-457f-8877-607fde9266e9\") " pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.833939 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.833852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:29.974468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:29.974441 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfb67b584-mh89d"] Aug 04 18:27:29.976716 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:27:29.976690 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76101e4_4f7b_457f_8877_607fde9266e9.slice/crio-171033b3177fc857a39308c6d1c29cc65ff2888d4c53bfda4123aff58133b937 WatchSource:0}: Error finding container 171033b3177fc857a39308c6d1c29cc65ff2888d4c53bfda4123aff58133b937: Status 404 returned error can't find the container with id 171033b3177fc857a39308c6d1c29cc65ff2888d4c53bfda4123aff58133b937 Aug 04 18:27:30.399343 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:30.399296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfb67b584-mh89d" event={"ID":"b76101e4-4f7b-457f-8877-607fde9266e9","Type":"ContainerStarted","Data":"9032b54cf94be94df669c742265489ae042675699b4fc5fc794e1427e51b3702"} Aug 04 18:27:30.399540 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:30.399395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfb67b584-mh89d" event={"ID":"b76101e4-4f7b-457f-8877-607fde9266e9","Type":"ContainerStarted","Data":"171033b3177fc857a39308c6d1c29cc65ff2888d4c53bfda4123aff58133b937"} Aug 04 18:27:30.428117 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:30.428063 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dfb67b584-mh89d" podStartSLOduration=1.428046141 podStartE2EDuration="1.428046141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:30.425943923 +0000 UTC m=+1015.572760182" watchObservedRunningTime="2026-08-04 18:27:30.428046141 +0000 UTC m=+1015.574862400" Aug 04 18:27:39.834025 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:39.833942 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:39.834025 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:39.833991 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:39.838839 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:39.838817 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:40.453663 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:40.453635 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dfb67b584-mh89d" Aug 04 18:27:40.553905 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:27:40.553866 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc584c66d-6bjtf"] Aug 04 18:28:05.573553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.573487 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dc584c66d-6bjtf" podUID="ac50e80e-bcb2-4145-8665-f5c7a4d77a41" containerName="console" containerID="cri-o://eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5" gracePeriod=15 Aug 04 18:28:05.818889 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.818863 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:28:05.973791 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.973748 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-service-ca\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.973963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.973808 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-config\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.973963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.973842 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-trusted-ca-bundle\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.973963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.973882 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-oauth-config\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.973963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.973918 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kjng\" (UniqueName: \"kubernetes.io/projected/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-kube-api-access-4kjng\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.974173 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.973976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-serving-cert\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.974173 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-oauth-serving-cert\") pod \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\" (UID: \"ac50e80e-bcb2-4145-8665-f5c7a4d77a41\") " Aug 04 18:28:05.974272 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974222 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:28:05.974272 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974235 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-config" (OuterVolumeSpecName: "console-config") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:28:05.974411 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:28:05.974411 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974338 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:05.974411 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974396 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:05.974590 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.974568 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:28:05.976281 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.976247 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-kube-api-access-4kjng" (OuterVolumeSpecName: "kube-api-access-4kjng") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "kube-api-access-4kjng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:28:05.976452 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.976307 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:28:05.976608 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:05.976586 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac50e80e-bcb2-4145-8665-f5c7a4d77a41" (UID: "ac50e80e-bcb2-4145-8665-f5c7a4d77a41"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:28:06.075214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.075158 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:06.075214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.075200 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-oauth-serving-cert\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:06.075214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.075210 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-trusted-ca-bundle\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:06.075214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.075220 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-console-oauth-config\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:06.075214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.075228 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjng\" (UniqueName: \"kubernetes.io/projected/ac50e80e-bcb2-4145-8665-f5c7a4d77a41-kube-api-access-4kjng\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:28:06.593727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.593687 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac50e80e-bcb2-4145-8665-f5c7a4d77a41" containerID="eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5" exitCode=0 Aug 04 18:28:06.594171 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.593733 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc584c66d-6bjtf" Aug 04 18:28:06.594171 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.593749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc584c66d-6bjtf" event={"ID":"ac50e80e-bcb2-4145-8665-f5c7a4d77a41","Type":"ContainerDied","Data":"eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5"} Aug 04 18:28:06.594171 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.593796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc584c66d-6bjtf" event={"ID":"ac50e80e-bcb2-4145-8665-f5c7a4d77a41","Type":"ContainerDied","Data":"a06704606607113baff99a7724a2b6aef65bde62b07e334dcabbf2db1c43061e"} Aug 04 18:28:06.594171 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.593817 2575 scope.go:117] "RemoveContainer" containerID="eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5" Aug 04 18:28:06.603225 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.603206 2575 scope.go:117] "RemoveContainer" containerID="eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5" Aug 04 18:28:06.603492 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:28:06.603474 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5\": container with ID starting with eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5 not found: ID does not exist" containerID="eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5" Aug 04 18:28:06.603555 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.603499 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5"} err="failed to get container status \"eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5\": rpc error: code = NotFound desc = could not find container \"eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5\": container with ID starting with eea72b8c4d5e9f8bd9ea5d3cc15a0d6ddf9f53a05eba03c4453096ebf2f2dca5 not found: ID does not exist" Aug 04 18:28:06.619605 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.619577 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc584c66d-6bjtf"] Aug 04 18:28:06.626418 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:06.626393 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc584c66d-6bjtf"] Aug 04 18:28:07.348882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:28:07.348849 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac50e80e-bcb2-4145-8665-f5c7a4d77a41" path="/var/lib/kubelet/pods/ac50e80e-bcb2-4145-8665-f5c7a4d77a41/volumes" Aug 04 18:30:35.365315 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:30:35.365281 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:30:35.370076 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:30:35.370053 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:31:48.158464 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.158386 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2"] Aug 04 18:31:48.160823 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.158643 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" podUID="6ba55a57-860e-4efc-b0c1-173f462a25a4" containerName="manager" containerID="cri-o://495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73" gracePeriod=10 Aug 04 18:31:48.412472 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.412411 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:31:48.540094 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.540063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvwdn\" (UniqueName: \"kubernetes.io/projected/6ba55a57-860e-4efc-b0c1-173f462a25a4-kube-api-access-wvwdn\") pod \"6ba55a57-860e-4efc-b0c1-173f462a25a4\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " Aug 04 18:31:48.540240 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.540168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ba55a57-860e-4efc-b0c1-173f462a25a4-extensions-socket-volume\") pod \"6ba55a57-860e-4efc-b0c1-173f462a25a4\" (UID: \"6ba55a57-860e-4efc-b0c1-173f462a25a4\") " Aug 04 18:31:48.540559 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.540535 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba55a57-860e-4efc-b0c1-173f462a25a4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ba55a57-860e-4efc-b0c1-173f462a25a4" (UID: "6ba55a57-860e-4efc-b0c1-173f462a25a4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:31:48.542488 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.542461 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba55a57-860e-4efc-b0c1-173f462a25a4-kube-api-access-wvwdn" (OuterVolumeSpecName: "kube-api-access-wvwdn") pod "6ba55a57-860e-4efc-b0c1-173f462a25a4" (UID: "6ba55a57-860e-4efc-b0c1-173f462a25a4"). InnerVolumeSpecName "kube-api-access-wvwdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:31:48.641073 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.641029 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ba55a57-860e-4efc-b0c1-173f462a25a4-extensions-socket-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:31:48.641073 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.641068 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvwdn\" (UniqueName: \"kubernetes.io/projected/6ba55a57-860e-4efc-b0c1-173f462a25a4-kube-api-access-wvwdn\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:31:48.755652 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.755622 2575 generic.go:356] "Generic (PLEG): container finished" podID="6ba55a57-860e-4efc-b0c1-173f462a25a4" containerID="495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73" exitCode=0 Aug 04 18:31:48.755797 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.755669 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" Aug 04 18:31:48.755797 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.755710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" event={"ID":"6ba55a57-860e-4efc-b0c1-173f462a25a4","Type":"ContainerDied","Data":"495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73"} Aug 04 18:31:48.755797 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.755766 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2" event={"ID":"6ba55a57-860e-4efc-b0c1-173f462a25a4","Type":"ContainerDied","Data":"8e6a66e1316af4edb8ae8ac3908324aa1251545bd80fe8fe8d918b64c6fed676"} Aug 04 18:31:48.755797 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.755794 2575 scope.go:117] "RemoveContainer" containerID="495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73" Aug 04 18:31:48.766663 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.766644 2575 scope.go:117] "RemoveContainer" containerID="495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73" Aug 04 18:31:48.766915 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:31:48.766895 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73\": container with ID starting with 495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73 not found: ID does not exist" containerID="495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73" Aug 04 18:31:48.766980 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.766929 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73"} err="failed to get container status \"495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73\": rpc error: code = NotFound desc = could not find container \"495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73\": container with ID starting with 495ceaf89b18936f69973599ad29454763c54f81abecb488fd19815854b14f73 not found: ID does not exist" Aug 04 18:31:48.780015 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.779991 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2"] Aug 04 18:31:48.784630 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:48.784609 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-vpfx2"] Aug 04 18:31:49.350311 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:31:49.350275 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba55a57-860e-4efc-b0c1-173f462a25a4" path="/var/lib/kubelet/pods/6ba55a57-860e-4efc-b0c1-173f462a25a4/volumes" Aug 04 18:32:48.898970 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.898939 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm"] Aug 04 18:32:48.899349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.899280 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba55a57-860e-4efc-b0c1-173f462a25a4" containerName="manager" Aug 04 18:32:48.899349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.899291 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba55a57-860e-4efc-b0c1-173f462a25a4" containerName="manager" Aug 04 18:32:48.899349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.899298 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac50e80e-bcb2-4145-8665-f5c7a4d77a41" containerName="console" Aug 04 18:32:48.899349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.899304 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac50e80e-bcb2-4145-8665-f5c7a4d77a41" containerName="console" Aug 04 18:32:48.899519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.899383 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac50e80e-bcb2-4145-8665-f5c7a4d77a41" containerName="console" Aug 04 18:32:48.899519 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.899394 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba55a57-860e-4efc-b0c1-173f462a25a4" containerName="manager" Aug 04 18:32:48.903592 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.903576 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:48.907034 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.907012 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rrnp5\"" Aug 04 18:32:48.908155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.908138 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 18:32:48.908251 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.908140 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 18:32:48.915781 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.915760 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm"] Aug 04 18:32:48.956504 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.956475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a098735-4280-4110-abaf-20192fa1dfea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-sjbkm\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:48.956645 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:48.956515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzt9\" (UniqueName: \"kubernetes.io/projected/3a098735-4280-4110-abaf-20192fa1dfea-kube-api-access-mqzt9\") pod \"kuadrant-operator-controller-manager-68cf589589-sjbkm\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:49.056997 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:49.056973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a098735-4280-4110-abaf-20192fa1dfea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-sjbkm\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:49.057221 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:49.057012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzt9\" (UniqueName: \"kubernetes.io/projected/3a098735-4280-4110-abaf-20192fa1dfea-kube-api-access-mqzt9\") pod \"kuadrant-operator-controller-manager-68cf589589-sjbkm\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:49.057443 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:49.057405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a098735-4280-4110-abaf-20192fa1dfea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-sjbkm\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:49.066988 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:49.066968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzt9\" (UniqueName: \"kubernetes.io/projected/3a098735-4280-4110-abaf-20192fa1dfea-kube-api-access-mqzt9\") pod \"kuadrant-operator-controller-manager-68cf589589-sjbkm\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:49.215032 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:49.215004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:49.364271 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:49.364244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm"] Aug 04 18:32:49.365713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:32:49.365689 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a098735_4280_4110_abaf_20192fa1dfea.slice/crio-3450806106ea4240ba07ab2c1c5728a83eef4f5f7ac4ee3528be6c9ddfd89872 WatchSource:0}: Error finding container 3450806106ea4240ba07ab2c1c5728a83eef4f5f7ac4ee3528be6c9ddfd89872: Status 404 returned error can't find the container with id 3450806106ea4240ba07ab2c1c5728a83eef4f5f7ac4ee3528be6c9ddfd89872 Aug 04 18:32:50.074689 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:50.074653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" event={"ID":"3a098735-4280-4110-abaf-20192fa1dfea","Type":"ContainerStarted","Data":"411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91"} Aug 04 18:32:50.075097 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:50.074728 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" event={"ID":"3a098735-4280-4110-abaf-20192fa1dfea","Type":"ContainerStarted","Data":"3450806106ea4240ba07ab2c1c5728a83eef4f5f7ac4ee3528be6c9ddfd89872"} Aug 04 18:32:50.075097 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:50.074818 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:32:50.112455 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:32:50.112402 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" podStartSLOduration=2.112386118 podStartE2EDuration="2.112386118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:50.110478165 +0000 UTC m=+1335.257294446" watchObservedRunningTime="2026-08-04 18:32:50.112386118 +0000 UTC m=+1335.259202378" Aug 04 18:33:01.082450 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:33:01.082416 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:35:35.406761 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:35:35.406735 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:35:35.414000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:35:35.413977 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:40:35.443488 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:35.443456 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:40:35.450222 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:35.450198 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:40:41.992715 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:41.992672 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c"] Aug 04 18:40:41.996136 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:41.996097 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:41.999819 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:41.999796 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9756c1e3-dockercfg-rbdv5\"" Aug 04 18:40:42.000630 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.000611 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9756c1e3\"" Aug 04 18:40:42.014084 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.014061 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c"] Aug 04 18:40:42.087718 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.087682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjbn\" (UniqueName: \"kubernetes.io/projected/e80995ad-2a0d-42f3-a1bf-e23930ae8209-kube-api-access-8fjbn\") pod \"payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.087916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.087761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80995ad-2a0d-42f3-a1bf-e23930ae8209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.188750 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.188715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80995ad-2a0d-42f3-a1bf-e23930ae8209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.188954 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.188777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjbn\" (UniqueName: \"kubernetes.io/projected/e80995ad-2a0d-42f3-a1bf-e23930ae8209-kube-api-access-8fjbn\") pod \"payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.189533 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.189510 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80995ad-2a0d-42f3-a1bf-e23930ae8209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.201214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.201184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjbn\" (UniqueName: \"kubernetes.io/projected/e80995ad-2a0d-42f3-a1bf-e23930ae8209-kube-api-access-8fjbn\") pod \"payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.306127 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.306035 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:42.447792 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.447765 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c"] Aug 04 18:40:42.449756 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:40:42.449725 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80995ad_2a0d_42f3_a1bf_e23930ae8209.slice/crio-cd41fd2e0bd35951f2a72165076991f5c5479d69abb29feeb30c00be22ff2c0a WatchSource:0}: Error finding container cd41fd2e0bd35951f2a72165076991f5c5479d69abb29feeb30c00be22ff2c0a: Status 404 returned error can't find the container with id cd41fd2e0bd35951f2a72165076991f5c5479d69abb29feeb30c00be22ff2c0a Aug 04 18:40:42.451645 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.451627 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:40:42.553609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:42.553568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" event={"ID":"e80995ad-2a0d-42f3-a1bf-e23930ae8209","Type":"ContainerStarted","Data":"cd41fd2e0bd35951f2a72165076991f5c5479d69abb29feeb30c00be22ff2c0a"} Aug 04 18:40:45.572628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:45.572590 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" event={"ID":"e80995ad-2a0d-42f3-a1bf-e23930ae8209","Type":"ContainerStarted","Data":"f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d"} Aug 04 18:40:45.573082 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:45.572728 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:40:45.599212 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:45.599161 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" podStartSLOduration=2.070683576 podStartE2EDuration="4.599146231s" totalImagesPullingTime="2.528462655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:40:45.597113117 +0000 UTC m=+1810.743929387" watchObservedRunningTime="2026-08-04 18:40:45.599146231 +0000 UTC m=+1810.745962491" Aug 04 18:40:56.581061 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:40:56.581028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:41:11.817767 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:11.817732 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c"] Aug 04 18:41:11.818599 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:11.817923 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" podUID="e80995ad-2a0d-42f3-a1bf-e23930ae8209" containerName="payload-pre-processing" containerID="cri-o://f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d" gracePeriod=30 Aug 04 18:41:12.066319 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.066291 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:41:12.162978 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.162894 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fjbn\" (UniqueName: \"kubernetes.io/projected/e80995ad-2a0d-42f3-a1bf-e23930ae8209-kube-api-access-8fjbn\") pod \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " Aug 04 18:41:12.163123 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.163030 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80995ad-2a0d-42f3-a1bf-e23930ae8209-plugins-config-volume\") pod \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\" (UID: \"e80995ad-2a0d-42f3-a1bf-e23930ae8209\") " Aug 04 18:41:12.163475 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.163445 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80995ad-2a0d-42f3-a1bf-e23930ae8209-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e80995ad-2a0d-42f3-a1bf-e23930ae8209" (UID: "e80995ad-2a0d-42f3-a1bf-e23930ae8209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:12.165223 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.165201 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80995ad-2a0d-42f3-a1bf-e23930ae8209-kube-api-access-8fjbn" (OuterVolumeSpecName: "kube-api-access-8fjbn") pod "e80995ad-2a0d-42f3-a1bf-e23930ae8209" (UID: "e80995ad-2a0d-42f3-a1bf-e23930ae8209"). InnerVolumeSpecName "kube-api-access-8fjbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:12.264480 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.264444 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80995ad-2a0d-42f3-a1bf-e23930ae8209-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.264480 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.264472 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjbn\" (UniqueName: \"kubernetes.io/projected/e80995ad-2a0d-42f3-a1bf-e23930ae8209-kube-api-access-8fjbn\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:12.713645 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.713608 2575 generic.go:356] "Generic (PLEG): container finished" podID="e80995ad-2a0d-42f3-a1bf-e23930ae8209" containerID="f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d" exitCode=0 Aug 04 18:41:12.713839 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.713659 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" Aug 04 18:41:12.713839 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.713692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" event={"ID":"e80995ad-2a0d-42f3-a1bf-e23930ae8209","Type":"ContainerDied","Data":"f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d"} Aug 04 18:41:12.713839 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.713751 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c" event={"ID":"e80995ad-2a0d-42f3-a1bf-e23930ae8209","Type":"ContainerDied","Data":"cd41fd2e0bd35951f2a72165076991f5c5479d69abb29feeb30c00be22ff2c0a"} Aug 04 18:41:12.713839 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.713780 2575 scope.go:117] "RemoveContainer" containerID="f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d" Aug 04 18:41:12.723899 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.723867 2575 scope.go:117] "RemoveContainer" containerID="f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d" Aug 04 18:41:12.724160 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:41:12.724138 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d\": container with ID starting with f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d not found: ID does not exist" containerID="f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d" Aug 04 18:41:12.724229 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.724169 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d"} err="failed to get container status \"f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d\": rpc error: code = NotFound desc = could not find container \"f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d\": container with ID starting with f5e23cc8fae0fe163f337d23c23a1071a043ec91c652bc04f1d1cc3113c29d3d not found: ID does not exist" Aug 04 18:41:12.764517 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.764490 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c"] Aug 04 18:41:12.801213 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:12.801186 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9756c1e3-5c5ffdc444-l6q9c"] Aug 04 18:41:13.348759 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:13.348725 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80995ad-2a0d-42f3-a1bf-e23930ae8209" path="/var/lib/kubelet/pods/e80995ad-2a0d-42f3-a1bf-e23930ae8209/volumes" Aug 04 18:41:17.349479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.349452 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44"] Aug 04 18:41:17.349868 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.349850 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80995ad-2a0d-42f3-a1bf-e23930ae8209" containerName="payload-pre-processing" Aug 04 18:41:17.349937 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.349871 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80995ad-2a0d-42f3-a1bf-e23930ae8209" containerName="payload-pre-processing" Aug 04 18:41:17.349981 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.349944 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80995ad-2a0d-42f3-a1bf-e23930ae8209" containerName="payload-pre-processing" Aug 04 18:41:17.354532 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.354514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.361896 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.361876 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:41:17.362626 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.362608 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:41:17.362736 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.362719 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:41:17.363489 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.363471 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cf4512d1-serving-cert\"" Aug 04 18:41:17.372304 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.372279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44"] Aug 04 18:41:17.414250 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.414218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkpq\" (UniqueName: \"kubernetes.io/projected/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-kube-api-access-ljkpq\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.414429 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.414284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.414429 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.414316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-maas-api-tls\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.515349 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.515302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkpq\" (UniqueName: \"kubernetes.io/projected/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-kube-api-access-ljkpq\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.515547 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.515391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.515547 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.515414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-maas-api-tls\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.516046 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.516027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-openshift-service-ca\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.517965 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.517945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-maas-api-tls\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.543476 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.543448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkpq\" (UniqueName: \"kubernetes.io/projected/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-kube-api-access-ljkpq\") pod \"maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.664755 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.664663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:17.818773 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:17.818743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44"] Aug 04 18:41:17.818891 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:41:17.818822 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389f143c_594a_4e2e_8dd8_1c3b2ab4f4e2.slice/crio-860ab2279d01a92ef64c3cfa303bec11c0481e5f3e6b3648f8d5d7fc95d4055e WatchSource:0}: Error finding container 860ab2279d01a92ef64c3cfa303bec11c0481e5f3e6b3648f8d5d7fc95d4055e: Status 404 returned error can't find the container with id 860ab2279d01a92ef64c3cfa303bec11c0481e5f3e6b3648f8d5d7fc95d4055e Aug 04 18:41:18.439236 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.439186 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n"] Aug 04 18:41:18.443422 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.443398 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.447736 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.447706 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cf4512d1-dockercfg-dp8hb\"" Aug 04 18:41:18.447906 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.447763 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cf4512d1\"" Aug 04 18:41:18.478731 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.478693 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n"] Aug 04 18:41:18.515318 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.515280 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp"] Aug 04 18:41:18.518964 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.518934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.529011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.528596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk8ss\" (UniqueName: \"kubernetes.io/projected/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-kube-api-access-qk8ss\") pod \"payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.529011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.528707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.533479 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.533448 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp"] Aug 04 18:41:18.633174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.633135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk8ss\" (UniqueName: \"kubernetes.io/projected/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-kube-api-access-qk8ss\") pod \"payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.633406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.633220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a31c9f-8ae7-45f1-a928-b06d0e318124-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.633406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.633263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.633406 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.633323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nv9\" (UniqueName: \"kubernetes.io/projected/63a31c9f-8ae7-45f1-a928-b06d0e318124-kube-api-access-n2nv9\") pod \"payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.634318 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.634213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.654662 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.654629 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk8ss\" (UniqueName: \"kubernetes.io/projected/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-kube-api-access-qk8ss\") pod \"payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.733983 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.733886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a31c9f-8ae7-45f1-a928-b06d0e318124-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.734162 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.734015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nv9\" (UniqueName: \"kubernetes.io/projected/63a31c9f-8ae7-45f1-a928-b06d0e318124-kube-api-access-n2nv9\") pod \"payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.734813 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.734785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a31c9f-8ae7-45f1-a928-b06d0e318124-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.747144 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.747121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nv9\" (UniqueName: \"kubernetes.io/projected/63a31c9f-8ae7-45f1-a928-b06d0e318124-kube-api-access-n2nv9\") pod \"payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.752091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.752060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" event={"ID":"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2","Type":"ContainerStarted","Data":"860ab2279d01a92ef64c3cfa303bec11c0481e5f3e6b3648f8d5d7fc95d4055e"} Aug 04 18:41:18.758970 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.758424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:18.834997 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.834680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:18.962813 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:18.962261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n"] Aug 04 18:41:18.964099 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:41:18.964051 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cd4b9f_6f48_40ba_8594_6b6f988bcf2a.slice/crio-74c1c1935e64e22a5d823ffa7cb03501de99e470cf6ff45412e48d4779d844d3 WatchSource:0}: Error finding container 74c1c1935e64e22a5d823ffa7cb03501de99e470cf6ff45412e48d4779d844d3: Status 404 returned error can't find the container with id 74c1c1935e64e22a5d823ffa7cb03501de99e470cf6ff45412e48d4779d844d3 Aug 04 18:41:19.040108 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.040076 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp"] Aug 04 18:41:19.045126 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:41:19.045095 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a31c9f_8ae7_45f1_a928_b06d0e318124.slice/crio-8e4ee643716c1ca337f3041aaee9051d5d21c28d3ce93611573e7f53603e5423 WatchSource:0}: Error finding container 8e4ee643716c1ca337f3041aaee9051d5d21c28d3ce93611573e7f53603e5423: Status 404 returned error can't find the container with id 8e4ee643716c1ca337f3041aaee9051d5d21c28d3ce93611573e7f53603e5423 Aug 04 18:41:19.761092 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.761048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" event={"ID":"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a","Type":"ContainerStarted","Data":"89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1"} Aug 04 18:41:19.761582 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.761121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" event={"ID":"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a","Type":"ContainerStarted","Data":"74c1c1935e64e22a5d823ffa7cb03501de99e470cf6ff45412e48d4779d844d3"} Aug 04 18:41:19.761582 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.761171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:19.763002 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.762969 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" event={"ID":"63a31c9f-8ae7-45f1-a928-b06d0e318124","Type":"ContainerStarted","Data":"bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1"} Aug 04 18:41:19.763157 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.763025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" event={"ID":"63a31c9f-8ae7-45f1-a928-b06d0e318124","Type":"ContainerStarted","Data":"8e4ee643716c1ca337f3041aaee9051d5d21c28d3ce93611573e7f53603e5423"} Aug 04 18:41:19.763157 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.763067 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:19.782326 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.781764 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" podStartSLOduration=1.781746812 podStartE2EDuration="1.781746812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:19.779999399 +0000 UTC m=+1844.926815661" watchObservedRunningTime="2026-08-04 18:41:19.781746812 +0000 UTC m=+1844.928563075" Aug 04 18:41:19.804752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:19.804710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" podStartSLOduration=1.804694983 podStartE2EDuration="1.804694983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:19.804443426 +0000 UTC m=+1844.951259686" watchObservedRunningTime="2026-08-04 18:41:19.804694983 +0000 UTC m=+1844.951511242" Aug 04 18:41:20.773045 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:20.772994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" event={"ID":"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2","Type":"ContainerStarted","Data":"4361cb7b9f81a89ca43f0853fe38b8cc56ea007503de5b7b22ec831f1267feac"} Aug 04 18:41:20.773527 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:20.773405 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:26.790732 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:26.790700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:26.828386 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:26.828309 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" podStartSLOduration=7.400209775 podStartE2EDuration="9.828295273s" totalImagesPullingTime="2.428085498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:20.806293123 +0000 UTC m=+1845.953109383" watchObservedRunningTime="2026-08-04 18:41:26.828295273 +0000 UTC m=+1851.975111566" Aug 04 18:41:30.775101 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:30.775067 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:30.775556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:30.775158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:48.704502 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.704465 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44"] Aug 04 18:41:48.704934 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.704662 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" podUID="389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" containerName="maas-api" containerID="cri-o://4361cb7b9f81a89ca43f0853fe38b8cc56ea007503de5b7b22ec831f1267feac" gracePeriod=30 Aug 04 18:41:48.852927 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.852883 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp"] Aug 04 18:41:48.853179 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.853143 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" podUID="63a31c9f-8ae7-45f1-a928-b06d0e318124" containerName="payload-processing" containerID="cri-o://bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1" gracePeriod=30 Aug 04 18:41:48.863960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.863350 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n"] Aug 04 18:41:48.863960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.863580 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" podUID="89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" containerName="payload-pre-processing" containerID="cri-o://89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1" gracePeriod=30 Aug 04 18:41:48.932523 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.932488 2575 generic.go:356] "Generic (PLEG): container finished" podID="389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" containerID="4361cb7b9f81a89ca43f0853fe38b8cc56ea007503de5b7b22ec831f1267feac" exitCode=0 Aug 04 18:41:48.932690 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.932567 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" event={"ID":"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2","Type":"ContainerDied","Data":"4361cb7b9f81a89ca43f0853fe38b8cc56ea007503de5b7b22ec831f1267feac"} Aug 04 18:41:48.991765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.991737 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:48.994036 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.994012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-maas-api-tls\") pod \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " Aug 04 18:41:48.994152 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.994077 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljkpq\" (UniqueName: \"kubernetes.io/projected/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-kube-api-access-ljkpq\") pod \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " Aug 04 18:41:48.994191 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.994164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-openshift-service-ca\") pod \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\" (UID: \"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2\") " Aug 04 18:41:48.994753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.994710 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" (UID: "389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:48.996507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.996423 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" (UID: "389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:41:48.997080 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:48.997044 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-kube-api-access-ljkpq" (OuterVolumeSpecName: "kube-api-access-ljkpq") pod "389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" (UID: "389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2"). InnerVolumeSpecName "kube-api-access-ljkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:49.095245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.095209 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.095391 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.095249 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkpq\" (UniqueName: \"kubernetes.io/projected/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-kube-api-access-ljkpq\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.095391 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.095280 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.136110 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.136088 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:49.141000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.140982 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:49.195588 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.195555 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a31c9f-8ae7-45f1-a928-b06d0e318124-plugins-config-volume\") pod \"63a31c9f-8ae7-45f1-a928-b06d0e318124\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " Aug 04 18:41:49.195777 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.195620 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk8ss\" (UniqueName: \"kubernetes.io/projected/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-kube-api-access-qk8ss\") pod \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " Aug 04 18:41:49.195777 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.195671 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2nv9\" (UniqueName: \"kubernetes.io/projected/63a31c9f-8ae7-45f1-a928-b06d0e318124-kube-api-access-n2nv9\") pod \"63a31c9f-8ae7-45f1-a928-b06d0e318124\" (UID: \"63a31c9f-8ae7-45f1-a928-b06d0e318124\") " Aug 04 18:41:49.195900 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.195812 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-plugins-config-volume\") pod \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\" (UID: \"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a\") " Aug 04 18:41:49.196000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.195976 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a31c9f-8ae7-45f1-a928-b06d0e318124-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63a31c9f-8ae7-45f1-a928-b06d0e318124" (UID: "63a31c9f-8ae7-45f1-a928-b06d0e318124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:49.196169 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.196148 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a31c9f-8ae7-45f1-a928-b06d0e318124-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.196609 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.196586 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" (UID: "89cd4b9f-6f48-40ba-8594-6b6f988bcf2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:41:49.198355 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.198327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a31c9f-8ae7-45f1-a928-b06d0e318124-kube-api-access-n2nv9" (OuterVolumeSpecName: "kube-api-access-n2nv9") pod "63a31c9f-8ae7-45f1-a928-b06d0e318124" (UID: "63a31c9f-8ae7-45f1-a928-b06d0e318124"). InnerVolumeSpecName "kube-api-access-n2nv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:49.198530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.198409 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-kube-api-access-qk8ss" (OuterVolumeSpecName: "kube-api-access-qk8ss") pod "89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" (UID: "89cd4b9f-6f48-40ba-8594-6b6f988bcf2a"). InnerVolumeSpecName "kube-api-access-qk8ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:41:49.297288 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.297214 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.297288 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.297241 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk8ss\" (UniqueName: \"kubernetes.io/projected/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a-kube-api-access-qk8ss\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.297288 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.297251 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nv9\" (UniqueName: \"kubernetes.io/projected/63a31c9f-8ae7-45f1-a928-b06d0e318124-kube-api-access-n2nv9\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:41:49.939552 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.939511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" event={"ID":"389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2","Type":"ContainerDied","Data":"860ab2279d01a92ef64c3cfa303bec11c0481e5f3e6b3648f8d5d7fc95d4055e"} Aug 04 18:41:49.940035 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.939531 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44" Aug 04 18:41:49.940035 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.939608 2575 scope.go:117] "RemoveContainer" containerID="4361cb7b9f81a89ca43f0853fe38b8cc56ea007503de5b7b22ec831f1267feac" Aug 04 18:41:49.941167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.941042 2575 generic.go:356] "Generic (PLEG): container finished" podID="63a31c9f-8ae7-45f1-a928-b06d0e318124" containerID="bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1" exitCode=0 Aug 04 18:41:49.941167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.941123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" event={"ID":"63a31c9f-8ae7-45f1-a928-b06d0e318124","Type":"ContainerDied","Data":"bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1"} Aug 04 18:41:49.941167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.941147 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" Aug 04 18:41:49.941728 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.941180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp" event={"ID":"63a31c9f-8ae7-45f1-a928-b06d0e318124","Type":"ContainerDied","Data":"8e4ee643716c1ca337f3041aaee9051d5d21c28d3ce93611573e7f53603e5423"} Aug 04 18:41:49.942981 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.942955 2575 generic.go:356] "Generic (PLEG): container finished" podID="89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" containerID="89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1" exitCode=0 Aug 04 18:41:49.943109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.943009 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" Aug 04 18:41:49.943109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.943013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" event={"ID":"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a","Type":"ContainerDied","Data":"89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1"} Aug 04 18:41:49.943109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.943061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n" event={"ID":"89cd4b9f-6f48-40ba-8594-6b6f988bcf2a","Type":"ContainerDied","Data":"74c1c1935e64e22a5d823ffa7cb03501de99e470cf6ff45412e48d4779d844d3"} Aug 04 18:41:49.952025 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.952004 2575 scope.go:117] "RemoveContainer" containerID="bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1" Aug 04 18:41:49.960432 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.960414 2575 scope.go:117] "RemoveContainer" containerID="bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1" Aug 04 18:41:49.960784 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:41:49.960762 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1\": container with ID starting with bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1 not found: ID does not exist" containerID="bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1" Aug 04 18:41:49.960876 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.960792 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1"} err="failed to get container status \"bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1\": rpc error: code = NotFound desc = could not find container \"bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1\": container with ID starting with bdde683f158b1f157cc6a5f284fa9f55da214b86b212bb8de41038210189f9f1 not found: ID does not exist" Aug 04 18:41:49.960876 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.960810 2575 scope.go:117] "RemoveContainer" containerID="89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1" Aug 04 18:41:49.969872 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.969724 2575 scope.go:117] "RemoveContainer" containerID="89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1" Aug 04 18:41:49.970027 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:41:49.970007 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1\": container with ID starting with 89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1 not found: ID does not exist" containerID="89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1" Aug 04 18:41:49.970100 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.970036 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1"} err="failed to get container status \"89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1\": rpc error: code = NotFound desc = could not find container \"89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1\": container with ID starting with 89326deb589821be0cc61070f5e4abb2829d538d0b93f4c9a007d14c205c7ca1 not found: ID does not exist" Aug 04 18:41:49.974377 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.974336 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp"] Aug 04 18:41:49.981581 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.981556 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cf4512d1-5f99cc9bb5-nvtbp"] Aug 04 18:41:49.994315 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.994291 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n"] Aug 04 18:41:49.999632 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:49.999610 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cf4512d1-7b8456b484-4rp7n"] Aug 04 18:41:50.013064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:50.013041 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44"] Aug 04 18:41:50.018563 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:50.018537 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cf4512d1-669b97dbcf-btl44"] Aug 04 18:41:51.349104 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:51.349072 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" path="/var/lib/kubelet/pods/389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2/volumes" Aug 04 18:41:51.349691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:51.349670 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63a31c9f-8ae7-45f1-a928-b06d0e318124" path="/var/lib/kubelet/pods/63a31c9f-8ae7-45f1-a928-b06d0e318124/volumes" Aug 04 18:41:51.350001 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:51.349989 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" path="/var/lib/kubelet/pods/89cd4b9f-6f48-40ba-8594-6b6f988bcf2a/volumes" Aug 04 18:41:53.306125 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306090 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g"] Aug 04 18:41:53.306521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306458 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a31c9f-8ae7-45f1-a928-b06d0e318124" containerName="payload-processing" Aug 04 18:41:53.306521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306470 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a31c9f-8ae7-45f1-a928-b06d0e318124" containerName="payload-processing" Aug 04 18:41:53.306521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306492 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" containerName="payload-pre-processing" Aug 04 18:41:53.306521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306498 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" containerName="payload-pre-processing" Aug 04 18:41:53.306521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306508 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" containerName="maas-api" Aug 04 18:41:53.306521 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306514 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" containerName="maas-api" Aug 04 18:41:53.306712 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306565 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="63a31c9f-8ae7-45f1-a928-b06d0e318124" containerName="payload-processing" Aug 04 18:41:53.306712 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306577 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cd4b9f-6f48-40ba-8594-6b6f988bcf2a" containerName="payload-pre-processing" Aug 04 18:41:53.306712 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.306584 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="389f143c-594a-4e2e-8dd8-1c3b2ab4f4e2" containerName="maas-api" Aug 04 18:41:53.314533 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.314513 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.317498 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.317476 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3f6c154b-serving-cert\"" Aug 04 18:41:53.317616 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.317501 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:41:53.318495 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.318477 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:41:53.318582 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.318511 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:41:53.324214 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.324194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g"] Aug 04 18:41:53.333591 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.333567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.333688 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.333620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f578751f-90d2-4898-acdb-d5bee1def70f-openshift-service-ca\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.333688 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.333652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smn8z\" (UniqueName: \"kubernetes.io/projected/f578751f-90d2-4898-acdb-d5bee1def70f-kube-api-access-smn8z\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.435042 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.435002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.435213 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.435057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f578751f-90d2-4898-acdb-d5bee1def70f-openshift-service-ca\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.435213 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:41:53.435157 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-serving-cert: secret "maas-api-e2e-ait-3f6c154b-serving-cert" not found Aug 04 18:41:53.435213 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.435173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smn8z\" (UniqueName: \"kubernetes.io/projected/f578751f-90d2-4898-acdb-d5bee1def70f-kube-api-access-smn8z\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.435213 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:41:53.435215 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls podName:f578751f-90d2-4898-acdb-d5bee1def70f nodeName:}" failed. No retries permitted until 2026-08-04 18:41:53.935198534 +0000 UTC m=+1879.082014773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls") pod "maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" (UID: "f578751f-90d2-4898-acdb-d5bee1def70f") : secret "maas-api-e2e-ait-3f6c154b-serving-cert" not found Aug 04 18:41:53.435652 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.435636 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f578751f-90d2-4898-acdb-d5bee1def70f-openshift-service-ca\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.445855 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.445829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smn8z\" (UniqueName: \"kubernetes.io/projected/f578751f-90d2-4898-acdb-d5bee1def70f-kube-api-access-smn8z\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.940202 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.940173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:53.942802 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:53.942775 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls\") pod \"maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:54.183310 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.183273 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv"] Aug 04 18:41:54.187383 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.187352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.191551 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.191484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3f6c154b-dockercfg-p2fgl\"" Aug 04 18:41:54.191551 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.191489 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3f6c154b\"" Aug 04 18:41:54.206553 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.206529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv"] Aug 04 18:41:54.226042 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.226017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:54.242769 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.242743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzz6\" (UniqueName: \"kubernetes.io/projected/6b764f02-5d62-4149-b9f6-ab30a1e9a020-kube-api-access-gdzz6\") pod \"payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.242892 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.242786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b764f02-5d62-4149-b9f6-ab30a1e9a020-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.280385 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.279783 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74"] Aug 04 18:41:54.298309 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.297717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.300027 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.299852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74"] Aug 04 18:41:54.343618 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.343506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864ec90a-9165-4a7b-b504-7ff16c551d32-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.343618 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.343602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzz6\" (UniqueName: \"kubernetes.io/projected/6b764f02-5d62-4149-b9f6-ab30a1e9a020-kube-api-access-gdzz6\") pod \"payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.344104 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.343648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b764f02-5d62-4149-b9f6-ab30a1e9a020-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.344104 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.343693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bxn\" (UniqueName: \"kubernetes.io/projected/864ec90a-9165-4a7b-b504-7ff16c551d32-kube-api-access-b9bxn\") pod \"payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.344538 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.344513 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b764f02-5d62-4149-b9f6-ab30a1e9a020-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.363549 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.363516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzz6\" (UniqueName: \"kubernetes.io/projected/6b764f02-5d62-4149-b9f6-ab30a1e9a020-kube-api-access-gdzz6\") pod \"payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.394260 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.394233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g"] Aug 04 18:41:54.396034 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:41:54.396004 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf578751f_90d2_4898_acdb_d5bee1def70f.slice/crio-3eed79fb919e14e6a09ee9bed09b2851602bfea2fc285591386e5b94533f657b WatchSource:0}: Error finding container 3eed79fb919e14e6a09ee9bed09b2851602bfea2fc285591386e5b94533f657b: Status 404 returned error can't find the container with id 3eed79fb919e14e6a09ee9bed09b2851602bfea2fc285591386e5b94533f657b Aug 04 18:41:54.445111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.445045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bxn\" (UniqueName: \"kubernetes.io/projected/864ec90a-9165-4a7b-b504-7ff16c551d32-kube-api-access-b9bxn\") pod \"payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.445239 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.445113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864ec90a-9165-4a7b-b504-7ff16c551d32-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.445795 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.445774 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864ec90a-9165-4a7b-b504-7ff16c551d32-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.455469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.455448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bxn\" (UniqueName: \"kubernetes.io/projected/864ec90a-9165-4a7b-b504-7ff16c551d32-kube-api-access-b9bxn\") pod \"payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.498105 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.498063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.613528 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.613482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.654552 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.654525 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv"] Aug 04 18:41:54.656707 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:41:54.656680 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b764f02_5d62_4149_b9f6_ab30a1e9a020.slice/crio-82c76bdf60daada74f93d208e71385eb35bd9e29bc709303ee803027d6de78e9 WatchSource:0}: Error finding container 82c76bdf60daada74f93d208e71385eb35bd9e29bc709303ee803027d6de78e9: Status 404 returned error can't find the container with id 82c76bdf60daada74f93d208e71385eb35bd9e29bc709303ee803027d6de78e9 Aug 04 18:41:54.766636 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.766607 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74"] Aug 04 18:41:54.773728 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:41:54.773694 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864ec90a_9165_4a7b_b504_7ff16c551d32.slice/crio-502b1996e1a1883ddd330f8ae55a73d318f52449b054b657c1e31df96e821b10 WatchSource:0}: Error finding container 502b1996e1a1883ddd330f8ae55a73d318f52449b054b657c1e31df96e821b10: Status 404 returned error can't find the container with id 502b1996e1a1883ddd330f8ae55a73d318f52449b054b657c1e31df96e821b10 Aug 04 18:41:54.976610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.976522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" event={"ID":"6b764f02-5d62-4149-b9f6-ab30a1e9a020","Type":"ContainerStarted","Data":"4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5"} Aug 04 18:41:54.976610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.976585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" event={"ID":"6b764f02-5d62-4149-b9f6-ab30a1e9a020","Type":"ContainerStarted","Data":"82c76bdf60daada74f93d208e71385eb35bd9e29bc709303ee803027d6de78e9"} Aug 04 18:41:54.976843 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.976615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:41:54.978340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.978311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" event={"ID":"f578751f-90d2-4898-acdb-d5bee1def70f","Type":"ContainerStarted","Data":"4cef570f1bc98109b5a9a6e05dd044447e2c2e06fdba1d1c3e020e66b03f3342"} Aug 04 18:41:54.978500 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.978382 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" event={"ID":"f578751f-90d2-4898-acdb-d5bee1def70f","Type":"ContainerStarted","Data":"3eed79fb919e14e6a09ee9bed09b2851602bfea2fc285591386e5b94533f657b"} Aug 04 18:41:54.978500 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.978425 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:41:54.980403 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.980314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" event={"ID":"864ec90a-9165-4a7b-b504-7ff16c551d32","Type":"ContainerStarted","Data":"d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a"} Aug 04 18:41:54.980403 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.980378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" event={"ID":"864ec90a-9165-4a7b-b504-7ff16c551d32","Type":"ContainerStarted","Data":"502b1996e1a1883ddd330f8ae55a73d318f52449b054b657c1e31df96e821b10"} Aug 04 18:41:54.980612 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.980479 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:41:54.994168 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:54.994090 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" podStartSLOduration=0.99407337 podStartE2EDuration="994.07337ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:54.99346614 +0000 UTC m=+1880.140282401" watchObservedRunningTime="2026-08-04 18:41:54.99407337 +0000 UTC m=+1880.140889631" Aug 04 18:41:55.027775 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:55.027716 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" podStartSLOduration=1.027702552 podStartE2EDuration="1.027702552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:55.024138725 +0000 UTC m=+1880.170955012" watchObservedRunningTime="2026-08-04 18:41:55.027702552 +0000 UTC m=+1880.174518835" Aug 04 18:41:55.054721 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:41:55.054672 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" podStartSLOduration=1.605579453 podStartE2EDuration="2.054655491s" totalImagesPullingTime="449.076038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:55.052571917 +0000 UTC m=+1880.199388177" watchObservedRunningTime="2026-08-04 18:41:55.054655491 +0000 UTC m=+1880.201471752" Aug 04 18:42:00.991949 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:00.991859 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:42:05.988666 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:05.988620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:42:05.989117 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:05.989033 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:42:34.076094 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.076060 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g"] Aug 04 18:42:34.078967 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.076284 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" podUID="f578751f-90d2-4898-acdb-d5bee1def70f" containerName="maas-api" containerID="cri-o://4cef570f1bc98109b5a9a6e05dd044447e2c2e06fdba1d1c3e020e66b03f3342" gracePeriod=30 Aug 04 18:42:34.200072 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.200025 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74"] Aug 04 18:42:34.200471 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.200440 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" podUID="864ec90a-9165-4a7b-b504-7ff16c551d32" containerName="payload-processing" containerID="cri-o://d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a" gracePeriod=30 Aug 04 18:42:34.209954 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.209920 2575 generic.go:356] "Generic (PLEG): container finished" podID="f578751f-90d2-4898-acdb-d5bee1def70f" containerID="4cef570f1bc98109b5a9a6e05dd044447e2c2e06fdba1d1c3e020e66b03f3342" exitCode=0 Aug 04 18:42:34.210533 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.210090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" event={"ID":"f578751f-90d2-4898-acdb-d5bee1def70f","Type":"ContainerDied","Data":"4cef570f1bc98109b5a9a6e05dd044447e2c2e06fdba1d1c3e020e66b03f3342"} Aug 04 18:42:34.212016 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.211977 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv"] Aug 04 18:42:34.212282 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.212231 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" podUID="6b764f02-5d62-4149-b9f6-ab30a1e9a020" containerName="payload-pre-processing" containerID="cri-o://4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5" gracePeriod=30 Aug 04 18:42:34.346973 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.346938 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:42:34.404995 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.404957 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smn8z\" (UniqueName: \"kubernetes.io/projected/f578751f-90d2-4898-acdb-d5bee1def70f-kube-api-access-smn8z\") pod \"f578751f-90d2-4898-acdb-d5bee1def70f\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " Aug 04 18:42:34.405154 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.405111 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f578751f-90d2-4898-acdb-d5bee1def70f-openshift-service-ca\") pod \"f578751f-90d2-4898-acdb-d5bee1def70f\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " Aug 04 18:42:34.405215 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.405150 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls\") pod \"f578751f-90d2-4898-acdb-d5bee1def70f\" (UID: \"f578751f-90d2-4898-acdb-d5bee1def70f\") " Aug 04 18:42:34.405537 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.405481 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f578751f-90d2-4898-acdb-d5bee1def70f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f578751f-90d2-4898-acdb-d5bee1def70f" (UID: "f578751f-90d2-4898-acdb-d5bee1def70f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:42:34.409397 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.409350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f578751f-90d2-4898-acdb-d5bee1def70f-kube-api-access-smn8z" (OuterVolumeSpecName: "kube-api-access-smn8z") pod "f578751f-90d2-4898-acdb-d5bee1def70f" (UID: "f578751f-90d2-4898-acdb-d5bee1def70f"). InnerVolumeSpecName "kube-api-access-smn8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:34.411201 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.411162 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f578751f-90d2-4898-acdb-d5bee1def70f" (UID: "f578751f-90d2-4898-acdb-d5bee1def70f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:42:34.485981 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.485954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:42:34.494556 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.494535 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:42:34.505904 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.505880 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b764f02-5d62-4149-b9f6-ab30a1e9a020-plugins-config-volume\") pod \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " Aug 04 18:42:34.506047 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.505921 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9bxn\" (UniqueName: \"kubernetes.io/projected/864ec90a-9165-4a7b-b504-7ff16c551d32-kube-api-access-b9bxn\") pod \"864ec90a-9165-4a7b-b504-7ff16c551d32\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " Aug 04 18:42:34.506047 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.505940 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864ec90a-9165-4a7b-b504-7ff16c551d32-plugins-config-volume\") pod \"864ec90a-9165-4a7b-b504-7ff16c551d32\" (UID: \"864ec90a-9165-4a7b-b504-7ff16c551d32\") " Aug 04 18:42:34.506047 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506024 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdzz6\" (UniqueName: \"kubernetes.io/projected/6b764f02-5d62-4149-b9f6-ab30a1e9a020-kube-api-access-gdzz6\") pod \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\" (UID: \"6b764f02-5d62-4149-b9f6-ab30a1e9a020\") " Aug 04 18:42:34.506471 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506308 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864ec90a-9165-4a7b-b504-7ff16c551d32-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "864ec90a-9165-4a7b-b504-7ff16c551d32" (UID: "864ec90a-9165-4a7b-b504-7ff16c551d32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:42:34.506471 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b764f02-5d62-4149-b9f6-ab30a1e9a020-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b764f02-5d62-4149-b9f6-ab30a1e9a020" (UID: "6b764f02-5d62-4149-b9f6-ab30a1e9a020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:42:34.506946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506861 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b764f02-5d62-4149-b9f6-ab30a1e9a020-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:34.506946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506886 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864ec90a-9165-4a7b-b504-7ff16c551d32-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:34.506946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506899 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f578751f-90d2-4898-acdb-d5bee1def70f-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:34.506946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506912 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f578751f-90d2-4898-acdb-d5bee1def70f-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:34.506946 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.506926 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smn8z\" (UniqueName: \"kubernetes.io/projected/f578751f-90d2-4898-acdb-d5bee1def70f-kube-api-access-smn8z\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:34.508892 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.508867 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b764f02-5d62-4149-b9f6-ab30a1e9a020-kube-api-access-gdzz6" (OuterVolumeSpecName: "kube-api-access-gdzz6") pod "6b764f02-5d62-4149-b9f6-ab30a1e9a020" (UID: "6b764f02-5d62-4149-b9f6-ab30a1e9a020"). InnerVolumeSpecName "kube-api-access-gdzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:34.508892 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.508874 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864ec90a-9165-4a7b-b504-7ff16c551d32-kube-api-access-b9bxn" (OuterVolumeSpecName: "kube-api-access-b9bxn") pod "864ec90a-9165-4a7b-b504-7ff16c551d32" (UID: "864ec90a-9165-4a7b-b504-7ff16c551d32"). InnerVolumeSpecName "kube-api-access-b9bxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:34.608058 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.607964 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9bxn\" (UniqueName: \"kubernetes.io/projected/864ec90a-9165-4a7b-b504-7ff16c551d32-kube-api-access-b9bxn\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:34.608058 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:34.607998 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdzz6\" (UniqueName: \"kubernetes.io/projected/6b764f02-5d62-4149-b9f6-ab30a1e9a020-kube-api-access-gdzz6\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:42:35.217325 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.217288 2575 generic.go:356] "Generic (PLEG): container finished" podID="864ec90a-9165-4a7b-b504-7ff16c551d32" containerID="d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a" exitCode=0 Aug 04 18:42:35.217766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.217332 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" Aug 04 18:42:35.217766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.217396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" event={"ID":"864ec90a-9165-4a7b-b504-7ff16c551d32","Type":"ContainerDied","Data":"d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a"} Aug 04 18:42:35.217766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.217448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74" event={"ID":"864ec90a-9165-4a7b-b504-7ff16c551d32","Type":"ContainerDied","Data":"502b1996e1a1883ddd330f8ae55a73d318f52449b054b657c1e31df96e821b10"} Aug 04 18:42:35.217766 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.217482 2575 scope.go:117] "RemoveContainer" containerID="d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a" Aug 04 18:42:35.218919 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.218837 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b764f02-5d62-4149-b9f6-ab30a1e9a020" containerID="4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5" exitCode=0 Aug 04 18:42:35.218919 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.218877 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" Aug 04 18:42:35.219041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.218924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" event={"ID":"6b764f02-5d62-4149-b9f6-ab30a1e9a020","Type":"ContainerDied","Data":"4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5"} Aug 04 18:42:35.219041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.218979 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv" event={"ID":"6b764f02-5d62-4149-b9f6-ab30a1e9a020","Type":"ContainerDied","Data":"82c76bdf60daada74f93d208e71385eb35bd9e29bc709303ee803027d6de78e9"} Aug 04 18:42:35.220554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.220524 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" Aug 04 18:42:35.220554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.220539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g" event={"ID":"f578751f-90d2-4898-acdb-d5bee1def70f","Type":"ContainerDied","Data":"3eed79fb919e14e6a09ee9bed09b2851602bfea2fc285591386e5b94533f657b"} Aug 04 18:42:35.227954 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.227936 2575 scope.go:117] "RemoveContainer" containerID="d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a" Aug 04 18:42:35.228286 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:42:35.228258 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a\": container with ID starting with d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a not found: ID does not exist" containerID="d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a" Aug 04 18:42:35.228420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.228298 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a"} err="failed to get container status \"d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a\": rpc error: code = NotFound desc = could not find container \"d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a\": container with ID starting with d0237db2442abc04eebd0b92c93d149bf29c4ba6b893619a95eb64ada12f0c4a not found: ID does not exist" Aug 04 18:42:35.228420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.228321 2575 scope.go:117] "RemoveContainer" containerID="4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5" Aug 04 18:42:35.237713 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.237694 2575 scope.go:117] "RemoveContainer" containerID="4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5" Aug 04 18:42:35.237955 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:42:35.237936 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5\": container with ID starting with 4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5 not found: ID does not exist" containerID="4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5" Aug 04 18:42:35.238011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.237963 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5"} err="failed to get container status \"4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5\": rpc error: code = NotFound desc = could not find container \"4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5\": container with ID starting with 4f16a70db9c301496b068ec17cf56761cde4b41f3184a5ada7859d547be973a5 not found: ID does not exist" Aug 04 18:42:35.238011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.237982 2575 scope.go:117] "RemoveContainer" containerID="4cef570f1bc98109b5a9a6e05dd044447e2c2e06fdba1d1c3e020e66b03f3342" Aug 04 18:42:35.247572 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.247542 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74"] Aug 04 18:42:35.250888 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.250866 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f6c154b-57bf5d848-qtx74"] Aug 04 18:42:35.265498 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.265467 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv"] Aug 04 18:42:35.269962 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.269937 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3f6c154b-76b8866c96-rrtnv"] Aug 04 18:42:35.281221 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.281191 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g"] Aug 04 18:42:35.287743 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.287717 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f6c154b-7f48bc7cf6-sqs4g"] Aug 04 18:42:35.348909 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.348834 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b764f02-5d62-4149-b9f6-ab30a1e9a020" path="/var/lib/kubelet/pods/6b764f02-5d62-4149-b9f6-ab30a1e9a020/volumes" Aug 04 18:42:35.349194 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.349181 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864ec90a-9165-4a7b-b504-7ff16c551d32" path="/var/lib/kubelet/pods/864ec90a-9165-4a7b-b504-7ff16c551d32/volumes" Aug 04 18:42:35.349548 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:35.349535 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f578751f-90d2-4898-acdb-d5bee1def70f" path="/var/lib/kubelet/pods/f578751f-90d2-4898-acdb-d5bee1def70f/volumes" Aug 04 18:42:40.464241 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464207 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q"] Aug 04 18:42:40.464747 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464726 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b764f02-5d62-4149-b9f6-ab30a1e9a020" containerName="payload-pre-processing" Aug 04 18:42:40.464799 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464751 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b764f02-5d62-4149-b9f6-ab30a1e9a020" containerName="payload-pre-processing" Aug 04 18:42:40.464799 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464790 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f578751f-90d2-4898-acdb-d5bee1def70f" containerName="maas-api" Aug 04 18:42:40.464882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464799 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f578751f-90d2-4898-acdb-d5bee1def70f" containerName="maas-api" Aug 04 18:42:40.464882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464818 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864ec90a-9165-4a7b-b504-7ff16c551d32" containerName="payload-processing" Aug 04 18:42:40.464882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464827 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="864ec90a-9165-4a7b-b504-7ff16c551d32" containerName="payload-processing" Aug 04 18:42:40.464996 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464915 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f578751f-90d2-4898-acdb-d5bee1def70f" containerName="maas-api" Aug 04 18:42:40.464996 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464928 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="864ec90a-9165-4a7b-b504-7ff16c551d32" containerName="payload-processing" Aug 04 18:42:40.464996 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.464939 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b764f02-5d62-4149-b9f6-ab30a1e9a020" containerName="payload-pre-processing" Aug 04 18:42:40.468312 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.468295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.471091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.471065 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:42:40.471216 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.471068 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:42:40.471848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.471833 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-16e32844-serving-cert\"" Aug 04 18:42:40.471943 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.471873 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:42:40.480786 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.480679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q"] Aug 04 18:42:40.560036 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.559998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.560229 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.560133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-openshift-service-ca\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.560229 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.560217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6hr\" (UniqueName: \"kubernetes.io/projected/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-kube-api-access-bq6hr\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.661170 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.661129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-openshift-service-ca\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.661399 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.661226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6hr\" (UniqueName: \"kubernetes.io/projected/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-kube-api-access-bq6hr\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.661399 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.661304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.661541 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:42:40.661437 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-serving-cert: secret "maas-api-e2e-derive-16e32844-serving-cert" not found Aug 04 18:42:40.661541 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:42:40.661504 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls podName:edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8 nodeName:}" failed. No retries permitted until 2026-08-04 18:42:41.161485802 +0000 UTC m=+1926.308302042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls") pod "maas-api-e2e-derive-16e32844-76bd54f646-qt22q" (UID: "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8") : secret "maas-api-e2e-derive-16e32844-serving-cert" not found Aug 04 18:42:40.661919 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.661897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-openshift-service-ca\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:40.672764 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:40.672738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6hr\" (UniqueName: \"kubernetes.io/projected/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-kube-api-access-bq6hr\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:41.165875 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.165840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:41.168517 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.168487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls\") pod \"maas-api-e2e-derive-16e32844-76bd54f646-qt22q\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:41.213776 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.213739 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg"] Aug 04 18:42:41.217569 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.217545 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf"] Aug 04 18:42:41.217706 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.217663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.219960 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.219938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-16e32844\"" Aug 04 18:42:41.220629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.220611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.222729 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.222709 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-16e32844-dockercfg-bjqzp\"" Aug 04 18:42:41.229426 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.229400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg"] Aug 04 18:42:41.233760 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.233733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf"] Aug 04 18:42:41.266801 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.266770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8cpj\" (UniqueName: \"kubernetes.io/projected/c69bfbaf-7628-4e25-acab-4e026aa5cab4-kube-api-access-s8cpj\") pod \"payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.266964 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.266825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bfbaf-7628-4e25-acab-4e026aa5cab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.266964 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.266858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58bj\" (UniqueName: \"kubernetes.io/projected/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-kube-api-access-g58bj\") pod \"payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.267042 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.266979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-plugins-config-volume\") pod \"payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.368408 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.368355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cpj\" (UniqueName: \"kubernetes.io/projected/c69bfbaf-7628-4e25-acab-4e026aa5cab4-kube-api-access-s8cpj\") pod \"payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.368554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.368419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bfbaf-7628-4e25-acab-4e026aa5cab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.368554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.368440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58bj\" (UniqueName: \"kubernetes.io/projected/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-kube-api-access-g58bj\") pod \"payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.368554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.368509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-plugins-config-volume\") pod \"payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.369143 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.369123 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-plugins-config-volume\") pod \"payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.369212 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.369163 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bfbaf-7628-4e25-acab-4e026aa5cab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.380352 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.380328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58bj\" (UniqueName: \"kubernetes.io/projected/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-kube-api-access-g58bj\") pod \"payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.380514 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.380392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cpj\" (UniqueName: \"kubernetes.io/projected/c69bfbaf-7628-4e25-acab-4e026aa5cab4-kube-api-access-s8cpj\") pod \"payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.382191 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.382173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:41.530290 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.530245 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q"] Aug 04 18:42:41.531434 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.531356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:41.531605 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:42:41.531580 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf1f1eb_8fd8_45fa_8347_1c35bc1f79f8.slice/crio-1c9951f822b20ddc25655cdda40a2303dab76614faa9d167c12ebb50cff0d4e8 WatchSource:0}: Error finding container 1c9951f822b20ddc25655cdda40a2303dab76614faa9d167c12ebb50cff0d4e8: Status 404 returned error can't find the container with id 1c9951f822b20ddc25655cdda40a2303dab76614faa9d167c12ebb50cff0d4e8 Aug 04 18:42:41.540422 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.540400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:41.679722 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.679690 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg"] Aug 04 18:42:41.681342 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:42:41.681313 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e2f04a_bdd9_4344_b178_df9b9721bdd9.slice/crio-2e8e2977090201bcf41eeb846a04f996e54b755f2e7fad625047de34d9bcb816 WatchSource:0}: Error finding container 2e8e2977090201bcf41eeb846a04f996e54b755f2e7fad625047de34d9bcb816: Status 404 returned error can't find the container with id 2e8e2977090201bcf41eeb846a04f996e54b755f2e7fad625047de34d9bcb816 Aug 04 18:42:41.718004 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:41.717976 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf"] Aug 04 18:42:41.723088 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:42:41.723049 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69bfbaf_7628_4e25_acab_4e026aa5cab4.slice/crio-45b8b2177e29def414cae210e53e23ac48c237f455f6e08551e6a49a075fe1f8 WatchSource:0}: Error finding container 45b8b2177e29def414cae210e53e23ac48c237f455f6e08551e6a49a075fe1f8: Status 404 returned error can't find the container with id 45b8b2177e29def414cae210e53e23ac48c237f455f6e08551e6a49a075fe1f8 Aug 04 18:42:42.262694 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.262650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" event={"ID":"c69bfbaf-7628-4e25-acab-4e026aa5cab4","Type":"ContainerStarted","Data":"ec89b359a4fd3d26d59b8730bab459d2b40e66f05aa664b4e985987ccef8528d"} Aug 04 18:42:42.262901 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.262715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" event={"ID":"c69bfbaf-7628-4e25-acab-4e026aa5cab4","Type":"ContainerStarted","Data":"45b8b2177e29def414cae210e53e23ac48c237f455f6e08551e6a49a075fe1f8"} Aug 04 18:42:42.262901 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.262779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:42:42.264602 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.264574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" event={"ID":"e7e2f04a-bdd9-4344-b178-df9b9721bdd9","Type":"ContainerStarted","Data":"4bde112ec1d13ef9875c0262e7bf78251052e2c269f787b3f5799c6a3a5e4ac4"} Aug 04 18:42:42.264752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.264624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" event={"ID":"e7e2f04a-bdd9-4344-b178-df9b9721bdd9","Type":"ContainerStarted","Data":"2e8e2977090201bcf41eeb846a04f996e54b755f2e7fad625047de34d9bcb816"} Aug 04 18:42:42.264752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.264668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:42.266216 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.266192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" event={"ID":"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8","Type":"ContainerStarted","Data":"f3abedabc64cfc06dff73d4756a58f11213ea5c01331dd7f2ecf8f456d5a4e95"} Aug 04 18:42:42.266340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.266239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" event={"ID":"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8","Type":"ContainerStarted","Data":"1c9951f822b20ddc25655cdda40a2303dab76614faa9d167c12ebb50cff0d4e8"} Aug 04 18:42:42.266340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.266270 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:42.285844 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.285776 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" podStartSLOduration=1.2857539 podStartE2EDuration="1.2857539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:42.282518044 +0000 UTC m=+1927.429334331" watchObservedRunningTime="2026-08-04 18:42:42.2857539 +0000 UTC m=+1927.432570162" Aug 04 18:42:42.306123 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.306072 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" podStartSLOduration=1.984928623 podStartE2EDuration="2.306056914s" totalImagesPullingTime="321.128291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:42.30488855 +0000 UTC m=+1927.451704834" watchObservedRunningTime="2026-08-04 18:42:42.306056914 +0000 UTC m=+1927.452873175" Aug 04 18:42:42.325314 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:42.325263 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" podStartSLOduration=1.325243165 podStartE2EDuration="1.325243165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:42.324235806 +0000 UTC m=+1927.471052068" watchObservedRunningTime="2026-08-04 18:42:42.325243165 +0000 UTC m=+1927.472059430" Aug 04 18:42:48.278380 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:48.278331 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:42:53.274851 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:53.274813 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:42:53.275251 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:42:53.274900 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:43:11.209783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.209742 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q"] Aug 04 18:43:11.210287 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.209993 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" podUID="edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" containerName="maas-api" containerID="cri-o://f3abedabc64cfc06dff73d4756a58f11213ea5c01331dd7f2ecf8f456d5a4e95" gracePeriod=30 Aug 04 18:43:11.287480 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.287443 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg"] Aug 04 18:43:11.288044 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.287991 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" podUID="e7e2f04a-bdd9-4344-b178-df9b9721bdd9" containerName="payload-processing" containerID="cri-o://4bde112ec1d13ef9875c0262e7bf78251052e2c269f787b3f5799c6a3a5e4ac4" gracePeriod=30 Aug 04 18:43:11.297133 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.296735 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf"] Aug 04 18:43:11.300611 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.297125 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" podUID="c69bfbaf-7628-4e25-acab-4e026aa5cab4" containerName="payload-pre-processing" containerID="cri-o://ec89b359a4fd3d26d59b8730bab459d2b40e66f05aa664b4e985987ccef8528d" gracePeriod=30 Aug 04 18:43:11.427113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.427063 2575 generic.go:356] "Generic (PLEG): container finished" podID="c69bfbaf-7628-4e25-acab-4e026aa5cab4" containerID="ec89b359a4fd3d26d59b8730bab459d2b40e66f05aa664b4e985987ccef8528d" exitCode=0 Aug 04 18:43:11.427404 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.427168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" event={"ID":"c69bfbaf-7628-4e25-acab-4e026aa5cab4","Type":"ContainerDied","Data":"ec89b359a4fd3d26d59b8730bab459d2b40e66f05aa664b4e985987ccef8528d"} Aug 04 18:43:11.429602 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.429506 2575 generic.go:356] "Generic (PLEG): container finished" podID="e7e2f04a-bdd9-4344-b178-df9b9721bdd9" containerID="4bde112ec1d13ef9875c0262e7bf78251052e2c269f787b3f5799c6a3a5e4ac4" exitCode=0 Aug 04 18:43:11.429602 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.429598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" event={"ID":"e7e2f04a-bdd9-4344-b178-df9b9721bdd9","Type":"ContainerDied","Data":"4bde112ec1d13ef9875c0262e7bf78251052e2c269f787b3f5799c6a3a5e4ac4"} Aug 04 18:43:11.431088 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.431062 2575 generic.go:356] "Generic (PLEG): container finished" podID="edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" containerID="f3abedabc64cfc06dff73d4756a58f11213ea5c01331dd7f2ecf8f456d5a4e95" exitCode=0 Aug 04 18:43:11.431236 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.431128 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" event={"ID":"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8","Type":"ContainerDied","Data":"f3abedabc64cfc06dff73d4756a58f11213ea5c01331dd7f2ecf8f456d5a4e95"} Aug 04 18:43:11.566027 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.565997 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:43:11.622732 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.622567 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:43:11.631284 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.631256 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:43:11.656942 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.656913 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-openshift-service-ca\") pod \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " Aug 04 18:43:11.657109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.656956 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-plugins-config-volume\") pod \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " Aug 04 18:43:11.657109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.656987 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8cpj\" (UniqueName: \"kubernetes.io/projected/c69bfbaf-7628-4e25-acab-4e026aa5cab4-kube-api-access-s8cpj\") pod \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " Aug 04 18:43:11.657109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657008 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g58bj\" (UniqueName: \"kubernetes.io/projected/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-kube-api-access-g58bj\") pod \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\" (UID: \"e7e2f04a-bdd9-4344-b178-df9b9721bdd9\") " Aug 04 18:43:11.657109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657059 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bfbaf-7628-4e25-acab-4e026aa5cab4-plugins-config-volume\") pod \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\" (UID: \"c69bfbaf-7628-4e25-acab-4e026aa5cab4\") " Aug 04 18:43:11.657109 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls\") pod \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " Aug 04 18:43:11.657385 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657112 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq6hr\" (UniqueName: \"kubernetes.io/projected/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-kube-api-access-bq6hr\") pod \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\" (UID: \"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8\") " Aug 04 18:43:11.657444 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657396 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" (UID: "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:43:11.657444 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657415 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7e2f04a-bdd9-4344-b178-df9b9721bdd9" (UID: "e7e2f04a-bdd9-4344-b178-df9b9721bdd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:43:11.657444 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.657429 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69bfbaf-7628-4e25-acab-4e026aa5cab4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c69bfbaf-7628-4e25-acab-4e026aa5cab4" (UID: "c69bfbaf-7628-4e25-acab-4e026aa5cab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:43:11.659536 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.659509 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69bfbaf-7628-4e25-acab-4e026aa5cab4-kube-api-access-s8cpj" (OuterVolumeSpecName: "kube-api-access-s8cpj") pod "c69bfbaf-7628-4e25-acab-4e026aa5cab4" (UID: "c69bfbaf-7628-4e25-acab-4e026aa5cab4"). InnerVolumeSpecName "kube-api-access-s8cpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:43:11.659680 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.659597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-kube-api-access-g58bj" (OuterVolumeSpecName: "kube-api-access-g58bj") pod "e7e2f04a-bdd9-4344-b178-df9b9721bdd9" (UID: "e7e2f04a-bdd9-4344-b178-df9b9721bdd9"). InnerVolumeSpecName "kube-api-access-g58bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:43:11.659903 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.659881 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" (UID: "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:43:11.660038 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.660014 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-kube-api-access-bq6hr" (OuterVolumeSpecName: "kube-api-access-bq6hr") pod "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" (UID: "edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8"). InnerVolumeSpecName "kube-api-access-bq6hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:43:11.758720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758634 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:11.758720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758667 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8cpj\" (UniqueName: \"kubernetes.io/projected/c69bfbaf-7628-4e25-acab-4e026aa5cab4-kube-api-access-s8cpj\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:11.758720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758680 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58bj\" (UniqueName: \"kubernetes.io/projected/e7e2f04a-bdd9-4344-b178-df9b9721bdd9-kube-api-access-g58bj\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:11.758720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758691 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69bfbaf-7628-4e25-acab-4e026aa5cab4-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:11.758720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758705 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:11.758720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758717 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6hr\" (UniqueName: \"kubernetes.io/projected/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-kube-api-access-bq6hr\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:11.759020 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:11.758730 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:43:12.437604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.437570 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" Aug 04 18:43:12.438054 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.437608 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q" event={"ID":"edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8","Type":"ContainerDied","Data":"1c9951f822b20ddc25655cdda40a2303dab76614faa9d167c12ebb50cff0d4e8"} Aug 04 18:43:12.438054 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.437663 2575 scope.go:117] "RemoveContainer" containerID="f3abedabc64cfc06dff73d4756a58f11213ea5c01331dd7f2ecf8f456d5a4e95" Aug 04 18:43:12.439080 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.439064 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" Aug 04 18:43:12.439162 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.439080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf" event={"ID":"c69bfbaf-7628-4e25-acab-4e026aa5cab4","Type":"ContainerDied","Data":"45b8b2177e29def414cae210e53e23ac48c237f455f6e08551e6a49a075fe1f8"} Aug 04 18:43:12.440703 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.440685 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" Aug 04 18:43:12.440813 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.440685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg" event={"ID":"e7e2f04a-bdd9-4344-b178-df9b9721bdd9","Type":"ContainerDied","Data":"2e8e2977090201bcf41eeb846a04f996e54b755f2e7fad625047de34d9bcb816"} Aug 04 18:43:12.448969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.448943 2575 scope.go:117] "RemoveContainer" containerID="ec89b359a4fd3d26d59b8730bab459d2b40e66f05aa664b4e985987ccef8528d" Aug 04 18:43:12.458304 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.458284 2575 scope.go:117] "RemoveContainer" containerID="4bde112ec1d13ef9875c0262e7bf78251052e2c269f787b3f5799c6a3a5e4ac4" Aug 04 18:43:12.489152 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.489099 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf"] Aug 04 18:43:12.494499 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.494475 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-16e32844-8467676576-w92lf"] Aug 04 18:43:12.505604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.505584 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg"] Aug 04 18:43:12.510317 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.510298 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-16e32844-6c5bbbd74b-7mlkg"] Aug 04 18:43:12.521470 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.521448 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q"] Aug 04 18:43:12.526606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:12.526580 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-16e32844-76bd54f646-qt22q"] Aug 04 18:43:13.352417 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:13.352379 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69bfbaf-7628-4e25-acab-4e026aa5cab4" path="/var/lib/kubelet/pods/c69bfbaf-7628-4e25-acab-4e026aa5cab4/volumes" Aug 04 18:43:13.352756 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:13.352743 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e2f04a-bdd9-4344-b178-df9b9721bdd9" path="/var/lib/kubelet/pods/e7e2f04a-bdd9-4344-b178-df9b9721bdd9/volumes" Aug 04 18:43:13.353093 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:43:13.353065 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" path="/var/lib/kubelet/pods/edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8/volumes" Aug 04 18:45:21.277506 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.277466 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw"] Aug 04 18:45:21.278002 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.277981 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" containerName="maas-api" Aug 04 18:45:21.278087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278006 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" containerName="maas-api" Aug 04 18:45:21.278087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278047 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e2f04a-bdd9-4344-b178-df9b9721bdd9" containerName="payload-processing" Aug 04 18:45:21.278087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278056 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e2f04a-bdd9-4344-b178-df9b9721bdd9" containerName="payload-processing" Aug 04 18:45:21.278087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278075 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69bfbaf-7628-4e25-acab-4e026aa5cab4" containerName="payload-pre-processing" Aug 04 18:45:21.278087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278085 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69bfbaf-7628-4e25-acab-4e026aa5cab4" containerName="payload-pre-processing" Aug 04 18:45:21.278282 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278199 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf1f1eb-8fd8-45fa-8347-1c35bc1f79f8" containerName="maas-api" Aug 04 18:45:21.278282 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278212 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69bfbaf-7628-4e25-acab-4e026aa5cab4" containerName="payload-pre-processing" Aug 04 18:45:21.278282 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.278223 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e2f04a-bdd9-4344-b178-df9b9721bdd9" containerName="payload-processing" Aug 04 18:45:21.281947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.281924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.285024 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.285002 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f11a0dab-serving-cert\"" Aug 04 18:45:21.285147 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.285093 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:45:21.285205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.285176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:45:21.285835 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.285817 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:45:21.292892 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.292863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw"] Aug 04 18:45:21.396879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.396849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmhf\" (UniqueName: \"kubernetes.io/projected/65df8ea5-1e43-4707-90be-291a5f57676a-kube-api-access-fpmhf\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.397038 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.396915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df8ea5-1e43-4707-90be-291a5f57676a-openshift-service-ca\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.397038 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.396966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.498227 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.498194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.498420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.498231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmhf\" (UniqueName: \"kubernetes.io/projected/65df8ea5-1e43-4707-90be-291a5f57676a-kube-api-access-fpmhf\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.498420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.498289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df8ea5-1e43-4707-90be-291a5f57676a-openshift-service-ca\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.498420 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:45:21.498333 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-serving-cert: secret "maas-api-e2e-mt-f11a0dab-serving-cert" not found Aug 04 18:45:21.498420 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:45:21.498419 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls podName:65df8ea5-1e43-4707-90be-291a5f57676a nodeName:}" failed. No retries permitted until 2026-08-04 18:45:21.998399286 +0000 UTC m=+2087.145215528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls") pod "maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" (UID: "65df8ea5-1e43-4707-90be-291a5f57676a") : secret "maas-api-e2e-mt-f11a0dab-serving-cert" not found Aug 04 18:45:21.498972 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.498950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df8ea5-1e43-4707-90be-291a5f57676a-openshift-service-ca\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:21.508299 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:21.508273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmhf\" (UniqueName: \"kubernetes.io/projected/65df8ea5-1e43-4707-90be-291a5f57676a-kube-api-access-fpmhf\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:22.002573 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.002539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:22.005153 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.005125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls\") pod \"maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:22.152146 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.152110 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297"] Aug 04 18:45:22.155603 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.155584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.158228 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.158208 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f11a0dab\"" Aug 04 18:45:22.169497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.169467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297"] Aug 04 18:45:22.194485 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.194459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:22.305817 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.305782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5nh\" (UniqueName: \"kubernetes.io/projected/61a50a7f-e69f-4092-bab7-66f724987002-kube-api-access-jv5nh\") pod \"payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.306455 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.305915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a50a7f-e69f-4092-bab7-66f724987002-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.309921 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.309894 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj"] Aug 04 18:45:22.313677 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.313656 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.316338 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.316319 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f11a0dab-dockercfg-t9j8t\"" Aug 04 18:45:22.325872 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.325852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj"] Aug 04 18:45:22.348898 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.348875 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw"] Aug 04 18:45:22.349679 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:45:22.349655 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65df8ea5_1e43_4707_90be_291a5f57676a.slice/crio-2eebfa61076f23de62ef3a5deeac60fb28ecd25cb4b44e0cf37ab2d36fda493d WatchSource:0}: Error finding container 2eebfa61076f23de62ef3a5deeac60fb28ecd25cb4b44e0cf37ab2d36fda493d: Status 404 returned error can't find the container with id 2eebfa61076f23de62ef3a5deeac60fb28ecd25cb4b44e0cf37ab2d36fda493d Aug 04 18:45:22.406535 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.406513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5nh\" (UniqueName: \"kubernetes.io/projected/61a50a7f-e69f-4092-bab7-66f724987002-kube-api-access-jv5nh\") pod \"payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.406660 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.406555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngs5\" (UniqueName: \"kubernetes.io/projected/74c54478-b1a7-43a9-bb7c-726677da60f3-kube-api-access-9ngs5\") pod \"payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.406729 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.406689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c54478-b1a7-43a9-bb7c-726677da60f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.406791 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.406761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a50a7f-e69f-4092-bab7-66f724987002-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.407440 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.407422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a50a7f-e69f-4092-bab7-66f724987002-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.426355 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.426338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5nh\" (UniqueName: \"kubernetes.io/projected/61a50a7f-e69f-4092-bab7-66f724987002-kube-api-access-jv5nh\") pod \"payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.466992 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.466967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:22.507976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.507946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c54478-b1a7-43a9-bb7c-726677da60f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.508099 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.508050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngs5\" (UniqueName: \"kubernetes.io/projected/74c54478-b1a7-43a9-bb7c-726677da60f3-kube-api-access-9ngs5\") pod \"payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.508656 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.508635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c54478-b1a7-43a9-bb7c-726677da60f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.521005 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.520983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngs5\" (UniqueName: \"kubernetes.io/projected/74c54478-b1a7-43a9-bb7c-726677da60f3-kube-api-access-9ngs5\") pod \"payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.609011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.608961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297"] Aug 04 18:45:22.611184 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:45:22.611156 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a50a7f_e69f_4092_bab7_66f724987002.slice/crio-28b3137f43d4d6ef84b5bb90d4e81fa2ff57f3a18cbaedc8534d4223860f3185 WatchSource:0}: Error finding container 28b3137f43d4d6ef84b5bb90d4e81fa2ff57f3a18cbaedc8534d4223860f3185: Status 404 returned error can't find the container with id 28b3137f43d4d6ef84b5bb90d4e81fa2ff57f3a18cbaedc8534d4223860f3185 Aug 04 18:45:22.626124 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.626101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:22.779796 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:22.779769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj"] Aug 04 18:45:22.789698 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:45:22.789670 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c54478_b1a7_43a9_bb7c_726677da60f3.slice/crio-9936252cfcbd29042ec6e0e880ef6a5438ec1c515ccae1bda0ef3dd83374e975 WatchSource:0}: Error finding container 9936252cfcbd29042ec6e0e880ef6a5438ec1c515ccae1bda0ef3dd83374e975: Status 404 returned error can't find the container with id 9936252cfcbd29042ec6e0e880ef6a5438ec1c515ccae1bda0ef3dd83374e975 Aug 04 18:45:23.122064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.121965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" event={"ID":"65df8ea5-1e43-4707-90be-291a5f57676a","Type":"ContainerStarted","Data":"435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d"} Aug 04 18:45:23.122064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.122032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" event={"ID":"65df8ea5-1e43-4707-90be-291a5f57676a","Type":"ContainerStarted","Data":"2eebfa61076f23de62ef3a5deeac60fb28ecd25cb4b44e0cf37ab2d36fda493d"} Aug 04 18:45:23.122064 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.122064 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:23.123459 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.123431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" event={"ID":"74c54478-b1a7-43a9-bb7c-726677da60f3","Type":"ContainerStarted","Data":"d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af"} Aug 04 18:45:23.123587 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.123474 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" event={"ID":"74c54478-b1a7-43a9-bb7c-726677da60f3","Type":"ContainerStarted","Data":"9936252cfcbd29042ec6e0e880ef6a5438ec1c515ccae1bda0ef3dd83374e975"} Aug 04 18:45:23.123587 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.123499 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:23.124818 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.124798 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" event={"ID":"61a50a7f-e69f-4092-bab7-66f724987002","Type":"ContainerStarted","Data":"c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5"} Aug 04 18:45:23.124928 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.124839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" event={"ID":"61a50a7f-e69f-4092-bab7-66f724987002","Type":"ContainerStarted","Data":"28b3137f43d4d6ef84b5bb90d4e81fa2ff57f3a18cbaedc8534d4223860f3185"} Aug 04 18:45:23.124928 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.124867 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:23.144050 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.144000 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" podStartSLOduration=1.736447292 podStartE2EDuration="2.143985296s" totalImagesPullingTime="407.538004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:23.140991825 +0000 UTC m=+2088.287808085" watchObservedRunningTime="2026-08-04 18:45:23.143985296 +0000 UTC m=+2088.290801555" Aug 04 18:45:23.163632 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.163572 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" podStartSLOduration=1.163558947 podStartE2EDuration="1.163558947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:23.160742618 +0000 UTC m=+2088.307558910" watchObservedRunningTime="2026-08-04 18:45:23.163558947 +0000 UTC m=+2088.310375240" Aug 04 18:45:23.181782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:23.181730 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" podStartSLOduration=1.181714864 podStartE2EDuration="1.181714864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:23.179251744 +0000 UTC m=+2088.326068016" watchObservedRunningTime="2026-08-04 18:45:23.181714864 +0000 UTC m=+2088.328531124" Aug 04 18:45:29.135505 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:29.135474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:45:34.131853 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:34.131820 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:45:34.132331 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:34.131908 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:45:35.477346 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:35.477321 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:45:35.484192 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:45:35.484170 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:46:09.790965 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:09.790926 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw"] Aug 04 18:46:09.791699 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:09.791162 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" podUID="65df8ea5-1e43-4707-90be-291a5f57676a" containerName="maas-api" containerID="cri-o://435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d" gracePeriod=30 Aug 04 18:46:09.868927 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:09.868893 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj"] Aug 04 18:46:09.869145 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:09.869116 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" podUID="74c54478-b1a7-43a9-bb7c-726677da60f3" containerName="payload-processing" containerID="cri-o://d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af" gracePeriod=30 Aug 04 18:46:09.882255 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:09.882218 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297"] Aug 04 18:46:09.882501 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:09.882470 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" podUID="61a50a7f-e69f-4092-bab7-66f724987002" containerName="payload-pre-processing" containerID="cri-o://c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5" gracePeriod=30 Aug 04 18:46:10.138420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.138392 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:46:10.178706 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.178680 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:46:10.185007 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.184983 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:46:10.239959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.239930 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpmhf\" (UniqueName: \"kubernetes.io/projected/65df8ea5-1e43-4707-90be-291a5f57676a-kube-api-access-fpmhf\") pod \"65df8ea5-1e43-4707-90be-291a5f57676a\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " Aug 04 18:46:10.240111 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.240004 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df8ea5-1e43-4707-90be-291a5f57676a-openshift-service-ca\") pod \"65df8ea5-1e43-4707-90be-291a5f57676a\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " Aug 04 18:46:10.240168 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.240106 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls\") pod \"65df8ea5-1e43-4707-90be-291a5f57676a\" (UID: \"65df8ea5-1e43-4707-90be-291a5f57676a\") " Aug 04 18:46:10.240401 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.240355 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65df8ea5-1e43-4707-90be-291a5f57676a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65df8ea5-1e43-4707-90be-291a5f57676a" (UID: "65df8ea5-1e43-4707-90be-291a5f57676a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:46:10.242287 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.242262 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65df8ea5-1e43-4707-90be-291a5f57676a-kube-api-access-fpmhf" (OuterVolumeSpecName: "kube-api-access-fpmhf") pod "65df8ea5-1e43-4707-90be-291a5f57676a" (UID: "65df8ea5-1e43-4707-90be-291a5f57676a"). InnerVolumeSpecName "kube-api-access-fpmhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:10.242418 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.242268 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65df8ea5-1e43-4707-90be-291a5f57676a" (UID: "65df8ea5-1e43-4707-90be-291a5f57676a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:46:10.341159 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341073 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a50a7f-e69f-4092-bab7-66f724987002-plugins-config-volume\") pod \"61a50a7f-e69f-4092-bab7-66f724987002\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " Aug 04 18:46:10.341159 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ngs5\" (UniqueName: \"kubernetes.io/projected/74c54478-b1a7-43a9-bb7c-726677da60f3-kube-api-access-9ngs5\") pod \"74c54478-b1a7-43a9-bb7c-726677da60f3\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " Aug 04 18:46:10.341481 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341176 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv5nh\" (UniqueName: \"kubernetes.io/projected/61a50a7f-e69f-4092-bab7-66f724987002-kube-api-access-jv5nh\") pod \"61a50a7f-e69f-4092-bab7-66f724987002\" (UID: \"61a50a7f-e69f-4092-bab7-66f724987002\") " Aug 04 18:46:10.341481 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341210 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c54478-b1a7-43a9-bb7c-726677da60f3-plugins-config-volume\") pod \"74c54478-b1a7-43a9-bb7c-726677da60f3\" (UID: \"74c54478-b1a7-43a9-bb7c-726677da60f3\") " Aug 04 18:46:10.341604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341461 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a50a7f-e69f-4092-bab7-66f724987002-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61a50a7f-e69f-4092-bab7-66f724987002" (UID: "61a50a7f-e69f-4092-bab7-66f724987002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:46:10.341604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341508 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65df8ea5-1e43-4707-90be-291a5f57676a-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.341604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341525 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmhf\" (UniqueName: \"kubernetes.io/projected/65df8ea5-1e43-4707-90be-291a5f57676a-kube-api-access-fpmhf\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.341604 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341539 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65df8ea5-1e43-4707-90be-291a5f57676a-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.341758 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.341615 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c54478-b1a7-43a9-bb7c-726677da60f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74c54478-b1a7-43a9-bb7c-726677da60f3" (UID: "74c54478-b1a7-43a9-bb7c-726677da60f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:46:10.343392 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.343353 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a50a7f-e69f-4092-bab7-66f724987002-kube-api-access-jv5nh" (OuterVolumeSpecName: "kube-api-access-jv5nh") pod "61a50a7f-e69f-4092-bab7-66f724987002" (UID: "61a50a7f-e69f-4092-bab7-66f724987002"). InnerVolumeSpecName "kube-api-access-jv5nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:10.343465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.343447 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c54478-b1a7-43a9-bb7c-726677da60f3-kube-api-access-9ngs5" (OuterVolumeSpecName: "kube-api-access-9ngs5") pod "74c54478-b1a7-43a9-bb7c-726677da60f3" (UID: "74c54478-b1a7-43a9-bb7c-726677da60f3"). InnerVolumeSpecName "kube-api-access-9ngs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:10.374045 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.374014 2575 generic.go:356] "Generic (PLEG): container finished" podID="61a50a7f-e69f-4092-bab7-66f724987002" containerID="c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5" exitCode=0 Aug 04 18:46:10.374176 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.374060 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" Aug 04 18:46:10.374176 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.374108 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" event={"ID":"61a50a7f-e69f-4092-bab7-66f724987002","Type":"ContainerDied","Data":"c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5"} Aug 04 18:46:10.374176 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.374174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297" event={"ID":"61a50a7f-e69f-4092-bab7-66f724987002","Type":"ContainerDied","Data":"28b3137f43d4d6ef84b5bb90d4e81fa2ff57f3a18cbaedc8534d4223860f3185"} Aug 04 18:46:10.374330 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.374204 2575 scope.go:117] "RemoveContainer" containerID="c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5" Aug 04 18:46:10.375516 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.375485 2575 generic.go:356] "Generic (PLEG): container finished" podID="65df8ea5-1e43-4707-90be-291a5f57676a" containerID="435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d" exitCode=0 Aug 04 18:46:10.375627 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.375522 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" Aug 04 18:46:10.375627 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.375568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" event={"ID":"65df8ea5-1e43-4707-90be-291a5f57676a","Type":"ContainerDied","Data":"435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d"} Aug 04 18:46:10.375745 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.375635 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw" event={"ID":"65df8ea5-1e43-4707-90be-291a5f57676a","Type":"ContainerDied","Data":"2eebfa61076f23de62ef3a5deeac60fb28ecd25cb4b44e0cf37ab2d36fda493d"} Aug 04 18:46:10.377179 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.377149 2575 generic.go:356] "Generic (PLEG): container finished" podID="74c54478-b1a7-43a9-bb7c-726677da60f3" containerID="d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af" exitCode=0 Aug 04 18:46:10.377281 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.377183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" event={"ID":"74c54478-b1a7-43a9-bb7c-726677da60f3","Type":"ContainerDied","Data":"d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af"} Aug 04 18:46:10.377281 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.377222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" event={"ID":"74c54478-b1a7-43a9-bb7c-726677da60f3","Type":"ContainerDied","Data":"9936252cfcbd29042ec6e0e880ef6a5438ec1c515ccae1bda0ef3dd83374e975"} Aug 04 18:46:10.377281 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.377279 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj" Aug 04 18:46:10.388085 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.388067 2575 scope.go:117] "RemoveContainer" containerID="c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5" Aug 04 18:46:10.388391 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:46:10.388340 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5\": container with ID starting with c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5 not found: ID does not exist" containerID="c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5" Aug 04 18:46:10.388501 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.388413 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5"} err="failed to get container status \"c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5\": rpc error: code = NotFound desc = could not find container \"c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5\": container with ID starting with c53e6b181d33ad4852a9718b961cbe07fbc158fe4df798ef1673f801d36276e5 not found: ID does not exist" Aug 04 18:46:10.388501 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.388437 2575 scope.go:117] "RemoveContainer" containerID="435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d" Aug 04 18:46:10.396651 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.396634 2575 scope.go:117] "RemoveContainer" containerID="435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d" Aug 04 18:46:10.396919 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:46:10.396896 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d\": container with ID starting with 435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d not found: ID does not exist" containerID="435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d" Aug 04 18:46:10.397003 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.396924 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d"} err="failed to get container status \"435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d\": rpc error: code = NotFound desc = could not find container \"435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d\": container with ID starting with 435e9b0a6c07786a5734030fe634943363f5f22d45d8c3a9df819c234d645b0d not found: ID does not exist" Aug 04 18:46:10.397003 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.396939 2575 scope.go:117] "RemoveContainer" containerID="d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af" Aug 04 18:46:10.404442 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.404428 2575 scope.go:117] "RemoveContainer" containerID="d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af" Aug 04 18:46:10.404702 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:46:10.404678 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af\": container with ID starting with d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af not found: ID does not exist" containerID="d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af" Aug 04 18:46:10.404748 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.404711 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af"} err="failed to get container status \"d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af\": rpc error: code = NotFound desc = could not find container \"d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af\": container with ID starting with d353b1bca87668d1ef6f1f37829b7ca2b4797fd80d57abe3245fd4159f4313af not found: ID does not exist" Aug 04 18:46:10.415174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.415153 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw"] Aug 04 18:46:10.419958 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.419937 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f11a0dab-76dd7ffb97-zd6xw"] Aug 04 18:46:10.441289 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.441227 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297"] Aug 04 18:46:10.444726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.443139 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngs5\" (UniqueName: \"kubernetes.io/projected/74c54478-b1a7-43a9-bb7c-726677da60f3-kube-api-access-9ngs5\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.444726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.443176 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv5nh\" (UniqueName: \"kubernetes.io/projected/61a50a7f-e69f-4092-bab7-66f724987002-kube-api-access-jv5nh\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.444726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.443193 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c54478-b1a7-43a9-bb7c-726677da60f3-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.444726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.443207 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a50a7f-e69f-4092-bab7-66f724987002-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:46:10.444726 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.444192 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f11a0dab-899544d64-mx297"] Aug 04 18:46:10.454278 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.454250 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj"] Aug 04 18:46:10.456900 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:10.456881 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f11a0dab-68db988d58-gp9pj"] Aug 04 18:46:11.348923 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:11.348887 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a50a7f-e69f-4092-bab7-66f724987002" path="/var/lib/kubelet/pods/61a50a7f-e69f-4092-bab7-66f724987002/volumes" Aug 04 18:46:11.349288 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:11.349266 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65df8ea5-1e43-4707-90be-291a5f57676a" path="/var/lib/kubelet/pods/65df8ea5-1e43-4707-90be-291a5f57676a/volumes" Aug 04 18:46:11.349615 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:11.349602 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c54478-b1a7-43a9-bb7c-726677da60f3" path="/var/lib/kubelet/pods/74c54478-b1a7-43a9-bb7c-726677da60f3/volumes" Aug 04 18:46:51.598933 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:51.598890 2575 generic.go:356] "Generic (PLEG): container finished" podID="261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e" containerID="e1a5b5611a820f036a255a34d9789e5b9458acc02b4b26309ba14441b55cb4ea" exitCode=1 Aug 04 18:46:51.599269 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:51.598964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" event={"ID":"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e","Type":"ContainerDied","Data":"e1a5b5611a820f036a255a34d9789e5b9458acc02b4b26309ba14441b55cb4ea"} Aug 04 18:46:51.599348 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:51.599330 2575 scope.go:117] "RemoveContainer" containerID="e1a5b5611a820f036a255a34d9789e5b9458acc02b4b26309ba14441b55cb4ea" Aug 04 18:46:52.605897 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:52.605862 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a098735-4280-4110-abaf-20192fa1dfea" containerID="411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91" exitCode=0 Aug 04 18:46:52.606307 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:52.605940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" event={"ID":"3a098735-4280-4110-abaf-20192fa1dfea","Type":"ContainerDied","Data":"411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91"} Aug 04 18:46:52.606398 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:52.606324 2575 scope.go:117] "RemoveContainer" containerID="411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91" Aug 04 18:46:52.608040 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:52.608020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" event={"ID":"261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e","Type":"ContainerStarted","Data":"8f371ba102cbc64a0dd30730e8e978ade48a0a62f227b85bb8b68dbc764fb89a"} Aug 04 18:46:52.608210 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:52.608192 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:46:53.614380 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:53.614333 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" event={"ID":"3a098735-4280-4110-abaf-20192fa1dfea","Type":"ContainerStarted","Data":"d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce"} Aug 04 18:46:53.614801 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:46:53.614535 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:47:03.616818 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:47:03.616786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84559d8974-dn8bf" Aug 04 18:47:04.621392 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:47:04.621342 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:47:12.365835 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:47:12.365793 2575 controller.go:195] "Failed to update lease" err="Put \"https://aeb9e022d670540fdaf7590f6abafc97-f9f04794ff9f3669.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-222.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Aug 04 18:48:13.109800 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.109758 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm"] Aug 04 18:48:13.110281 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.110007 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" containerID="cri-o://d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce" gracePeriod=10 Aug 04 18:48:13.354902 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.354879 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:48:13.493381 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.493339 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqzt9\" (UniqueName: \"kubernetes.io/projected/3a098735-4280-4110-abaf-20192fa1dfea-kube-api-access-mqzt9\") pod \"3a098735-4280-4110-abaf-20192fa1dfea\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " Aug 04 18:48:13.493573 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.493435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a098735-4280-4110-abaf-20192fa1dfea-extensions-socket-volume\") pod \"3a098735-4280-4110-abaf-20192fa1dfea\" (UID: \"3a098735-4280-4110-abaf-20192fa1dfea\") " Aug 04 18:48:13.493796 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.493772 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a098735-4280-4110-abaf-20192fa1dfea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3a098735-4280-4110-abaf-20192fa1dfea" (UID: "3a098735-4280-4110-abaf-20192fa1dfea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:48:13.495677 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.495655 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a098735-4280-4110-abaf-20192fa1dfea-kube-api-access-mqzt9" (OuterVolumeSpecName: "kube-api-access-mqzt9") pod "3a098735-4280-4110-abaf-20192fa1dfea" (UID: "3a098735-4280-4110-abaf-20192fa1dfea"). InnerVolumeSpecName "kube-api-access-mqzt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:48:13.594727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.594690 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a098735-4280-4110-abaf-20192fa1dfea-extensions-socket-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:48:13.594727 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:13.594718 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzt9\" (UniqueName: \"kubernetes.io/projected/3a098735-4280-4110-abaf-20192fa1dfea-kube-api-access-mqzt9\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:48:14.040455 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.040418 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a098735-4280-4110-abaf-20192fa1dfea" containerID="d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce" exitCode=0 Aug 04 18:48:14.040628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.040470 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" Aug 04 18:48:14.040628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.040496 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" event={"ID":"3a098735-4280-4110-abaf-20192fa1dfea","Type":"ContainerDied","Data":"d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce"} Aug 04 18:48:14.040628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.040547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm" event={"ID":"3a098735-4280-4110-abaf-20192fa1dfea","Type":"ContainerDied","Data":"3450806106ea4240ba07ab2c1c5728a83eef4f5f7ac4ee3528be6c9ddfd89872"} Aug 04 18:48:14.040628 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.040570 2575 scope.go:117] "RemoveContainer" containerID="d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce" Aug 04 18:48:14.052538 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.052519 2575 scope.go:117] "RemoveContainer" containerID="411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91" Aug 04 18:48:14.062530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.062512 2575 scope.go:117] "RemoveContainer" containerID="d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce" Aug 04 18:48:14.062809 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:48:14.062783 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce\": container with ID starting with d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce not found: ID does not exist" containerID="d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce" Aug 04 18:48:14.062864 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.062819 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce"} err="failed to get container status \"d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce\": rpc error: code = NotFound desc = could not find container \"d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce\": container with ID starting with d75b934e961746c7d66940da31977c68c6c7a40b7a6500c36423d687150085ce not found: ID does not exist" Aug 04 18:48:14.062864 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.062840 2575 scope.go:117] "RemoveContainer" containerID="411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91" Aug 04 18:48:14.063088 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:48:14.063072 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91\": container with ID starting with 411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91 not found: ID does not exist" containerID="411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91" Aug 04 18:48:14.063126 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.063093 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91"} err="failed to get container status \"411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91\": rpc error: code = NotFound desc = could not find container \"411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91\": container with ID starting with 411dc1a75d32eb7c81ce424113f09561362b55335f4dbbfcaf958e404fcbfb91 not found: ID does not exist" Aug 04 18:48:14.082081 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.082053 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm"] Aug 04 18:48:14.088679 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:14.088654 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sjbkm"] Aug 04 18:48:15.349467 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:48:15.349431 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a098735-4280-4110-abaf-20192fa1dfea" path="/var/lib/kubelet/pods/3a098735-4280-4110-abaf-20192fa1dfea/volumes" Aug 04 18:49:28.789957 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.789872 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99"] Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790296 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790311 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790323 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790331 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790347 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c54478-b1a7-43a9-bb7c-726677da60f3" containerName="payload-processing" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790354 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c54478-b1a7-43a9-bb7c-726677da60f3" containerName="payload-processing" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790389 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a50a7f-e69f-4092-bab7-66f724987002" containerName="payload-pre-processing" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790397 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a50a7f-e69f-4092-bab7-66f724987002" containerName="payload-pre-processing" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790404 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65df8ea5-1e43-4707-90be-291a5f57676a" containerName="maas-api" Aug 04 18:49:28.790431 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790410 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65df8ea5-1e43-4707-90be-291a5f57676a" containerName="maas-api" Aug 04 18:49:28.790753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790470 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" Aug 04 18:49:28.790753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790481 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a50a7f-e69f-4092-bab7-66f724987002" containerName="payload-pre-processing" Aug 04 18:49:28.790753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790488 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a098735-4280-4110-abaf-20192fa1dfea" containerName="manager" Aug 04 18:49:28.790753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790493 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65df8ea5-1e43-4707-90be-291a5f57676a" containerName="maas-api" Aug 04 18:49:28.790753 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.790498 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c54478-b1a7-43a9-bb7c-726677da60f3" containerName="payload-processing" Aug 04 18:49:28.793597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.793579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:28.799311 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.799285 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 18:49:28.802440 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:49:28.802414 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-rrnp5\" is forbidden: User \"system:node:ip-10-0-142-222.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-222.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rrnp5\"" type="*v1.Secret" Aug 04 18:49:28.802578 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:49:28.802505 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-222.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-222.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Aug 04 18:49:28.802752 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:49:28.802719 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-r6m99\" is forbidden: User \"system:node:ip-10-0-142-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-222.ec2.internal' and this object" podUID="27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:28.822412 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.822387 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99"] Aug 04 18:49:28.863507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.863484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:28.863620 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.863536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6l7\" (UniqueName: \"kubernetes.io/projected/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-kube-api-access-6t6l7\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:28.964495 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.964462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:28.964603 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.964517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6l7\" (UniqueName: \"kubernetes.io/projected/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-kube-api-access-6t6l7\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:28.964847 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:28.964827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:29.973742 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:49:29.973707 2575 projected.go:291] Couldn't get configMap kuadrant-system/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Aug 04 18:49:29.973742 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:49:29.973749 2575 projected.go:196] Error preparing data for projected volume kube-api-access-6t6l7 for pod kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99: failed to sync configmap cache: timed out waiting for the condition Aug 04 18:49:29.974151 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:49:29.973826 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-kube-api-access-6t6l7 podName:27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9 nodeName:}" failed. No retries permitted until 2026-08-04 18:49:30.473806602 +0000 UTC m=+2335.620622840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6t6l7" (UniqueName: "kubernetes.io/projected/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-kube-api-access-6t6l7") pod "kuadrant-operator-controller-manager-68cf589589-r6m99" (UID: "27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9") : failed to sync configmap cache: timed out waiting for the condition Aug 04 18:49:30.247771 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:30.247691 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rrnp5\"" Aug 04 18:49:30.364974 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:30.364948 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 18:49:30.478257 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:30.478217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6l7\" (UniqueName: \"kubernetes.io/projected/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-kube-api-access-6t6l7\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:30.481175 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:30.481152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6l7\" (UniqueName: \"kubernetes.io/projected/27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9-kube-api-access-6t6l7\") pod \"kuadrant-operator-controller-manager-68cf589589-r6m99\" (UID: \"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:30.605113 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:30.605053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:30.758617 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:30.758160 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99"] Aug 04 18:49:30.760713 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:49:30.760676 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b3f9a1_7f75_463c_bf41_9cdf6d0b77b9.slice/crio-86ffb29ceb2751f92e11dc95ac1fa0d79e3db957fd35cf20e3e864a233bd609d WatchSource:0}: Error finding container 86ffb29ceb2751f92e11dc95ac1fa0d79e3db957fd35cf20e3e864a233bd609d: Status 404 returned error can't find the container with id 86ffb29ceb2751f92e11dc95ac1fa0d79e3db957fd35cf20e3e864a233bd609d Aug 04 18:49:31.431248 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:31.431206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" event={"ID":"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9","Type":"ContainerStarted","Data":"0e602381576a570e7c67bf8049818afae6316d4c34b9dab9c1fb67d8af3702cd"} Aug 04 18:49:31.431671 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:31.431270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" event={"ID":"27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9","Type":"ContainerStarted","Data":"86ffb29ceb2751f92e11dc95ac1fa0d79e3db957fd35cf20e3e864a233bd609d"} Aug 04 18:49:31.431671 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:31.431292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:31.470623 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:31.470571 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" podStartSLOduration=3.470555315 podStartE2EDuration="3.470555315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:31.469853472 +0000 UTC m=+2336.616669736" watchObservedRunningTime="2026-08-04 18:49:31.470555315 +0000 UTC m=+2336.617371575" Aug 04 18:49:42.439258 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:42.439221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-r6m99" Aug 04 18:49:53.475598 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.475564 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh"] Aug 04 18:49:53.479254 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.479232 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.482052 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.482033 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:49:53.482958 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.482928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fee88ca8-serving-cert\"" Aug 04 18:49:53.483043 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.482967 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:49:53.483098 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.483049 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:49:53.497333 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.497308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh"] Aug 04 18:49:53.580775 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.580745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9402c05-7880-4112-b746-b89ca9a2ad44-maas-api-tls\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.580933 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.580793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgv2\" (UniqueName: \"kubernetes.io/projected/a9402c05-7880-4112-b746-b89ca9a2ad44-kube-api-access-tjgv2\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.580933 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.580859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9402c05-7880-4112-b746-b89ca9a2ad44-openshift-service-ca\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.681944 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.681902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgv2\" (UniqueName: \"kubernetes.io/projected/a9402c05-7880-4112-b746-b89ca9a2ad44-kube-api-access-tjgv2\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.682131 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.681964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9402c05-7880-4112-b746-b89ca9a2ad44-openshift-service-ca\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.682131 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.682012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9402c05-7880-4112-b746-b89ca9a2ad44-maas-api-tls\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.682740 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.682708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9402c05-7880-4112-b746-b89ca9a2ad44-openshift-service-ca\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.684612 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.684593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9402c05-7880-4112-b746-b89ca9a2ad44-maas-api-tls\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.691211 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.691186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgv2\" (UniqueName: \"kubernetes.io/projected/a9402c05-7880-4112-b746-b89ca9a2ad44-kube-api-access-tjgv2\") pod \"maas-api-e2e-mt-fee88ca8-798f476f69-pvszh\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.789947 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.789859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:53.953399 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.953341 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh"] Aug 04 18:49:53.956088 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:49:53.956049 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9402c05_7880_4112_b746_b89ca9a2ad44.slice/crio-cb17c3fde3289546e2ffc8c559fa2b27475b0ab9a67a8031f604291ebb661956 WatchSource:0}: Error finding container cb17c3fde3289546e2ffc8c559fa2b27475b0ab9a67a8031f604291ebb661956: Status 404 returned error can't find the container with id cb17c3fde3289546e2ffc8c559fa2b27475b0ab9a67a8031f604291ebb661956 Aug 04 18:49:53.957776 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:53.957759 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:49:54.394790 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.394740 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h"] Aug 04 18:49:54.399311 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.399287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.401610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.401589 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fee88ca8\"" Aug 04 18:49:54.409685 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.409660 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h"] Aug 04 18:49:54.479734 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.479699 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw"] Aug 04 18:49:54.483216 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.483196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.486802 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.486777 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fee88ca8-dockercfg-dbcff\"" Aug 04 18:49:54.487630 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.487606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4682b881-744d-45c2-9ca9-e90ceaa1c654-plugins-config-volume\") pod \"payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.487783 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.487767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlk42\" (UniqueName: \"kubernetes.io/projected/4682b881-744d-45c2-9ca9-e90ceaa1c654-kube-api-access-vlk42\") pod \"payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.540756 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.540721 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw"] Aug 04 18:49:54.551934 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.551903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" event={"ID":"a9402c05-7880-4112-b746-b89ca9a2ad44","Type":"ContainerStarted","Data":"cb17c3fde3289546e2ffc8c559fa2b27475b0ab9a67a8031f604291ebb661956"} Aug 04 18:49:54.588637 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.588599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbx8\" (UniqueName: \"kubernetes.io/projected/94facd6b-dc41-46f9-9aa5-8720ec291f05-kube-api-access-fnbx8\") pod \"payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.588811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.588721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk42\" (UniqueName: \"kubernetes.io/projected/4682b881-744d-45c2-9ca9-e90ceaa1c654-kube-api-access-vlk42\") pod \"payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.588811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.588757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94facd6b-dc41-46f9-9aa5-8720ec291f05-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.588811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.588784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4682b881-744d-45c2-9ca9-e90ceaa1c654-plugins-config-volume\") pod \"payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.589533 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.589517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4682b881-744d-45c2-9ca9-e90ceaa1c654-plugins-config-volume\") pod \"payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.600849 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.600827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk42\" (UniqueName: \"kubernetes.io/projected/4682b881-744d-45c2-9ca9-e90ceaa1c654-kube-api-access-vlk42\") pod \"payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.689540 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.689504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94facd6b-dc41-46f9-9aa5-8720ec291f05-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.689687 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.689593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbx8\" (UniqueName: \"kubernetes.io/projected/94facd6b-dc41-46f9-9aa5-8720ec291f05-kube-api-access-fnbx8\") pod \"payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.690193 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.690173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94facd6b-dc41-46f9-9aa5-8720ec291f05-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.701210 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.701188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbx8\" (UniqueName: \"kubernetes.io/projected/94facd6b-dc41-46f9-9aa5-8720ec291f05-kube-api-access-fnbx8\") pod \"payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.711972 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.711947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:54.794507 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.794479 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:54.868353 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.868319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h"] Aug 04 18:49:54.869333 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:49:54.869307 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4682b881_744d_45c2_9ca9_e90ceaa1c654.slice/crio-303f3957f68cc5ead782f2a815d09f14eeac300a4c54cc25c3898692c490009d WatchSource:0}: Error finding container 303f3957f68cc5ead782f2a815d09f14eeac300a4c54cc25c3898692c490009d: Status 404 returned error can't find the container with id 303f3957f68cc5ead782f2a815d09f14eeac300a4c54cc25c3898692c490009d Aug 04 18:49:54.938925 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:54.938902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw"] Aug 04 18:49:54.939954 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:49:54.939899 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94facd6b_dc41_46f9_9aa5_8720ec291f05.slice/crio-2ac8c8a1ef57497cdc485f2700bb4254a6225f16a38dca1418057fe19f77f8b3 WatchSource:0}: Error finding container 2ac8c8a1ef57497cdc485f2700bb4254a6225f16a38dca1418057fe19f77f8b3: Status 404 returned error can't find the container with id 2ac8c8a1ef57497cdc485f2700bb4254a6225f16a38dca1418057fe19f77f8b3 Aug 04 18:49:55.559806 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.559700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" event={"ID":"a9402c05-7880-4112-b746-b89ca9a2ad44","Type":"ContainerStarted","Data":"9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61"} Aug 04 18:49:55.559806 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.559775 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:49:55.561412 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.561348 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" event={"ID":"94facd6b-dc41-46f9-9aa5-8720ec291f05","Type":"ContainerStarted","Data":"e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0"} Aug 04 18:49:55.561597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.561437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" event={"ID":"94facd6b-dc41-46f9-9aa5-8720ec291f05","Type":"ContainerStarted","Data":"2ac8c8a1ef57497cdc485f2700bb4254a6225f16a38dca1418057fe19f77f8b3"} Aug 04 18:49:55.561597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.561477 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:49:55.563265 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.563240 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" event={"ID":"4682b881-744d-45c2-9ca9-e90ceaa1c654","Type":"ContainerStarted","Data":"e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa"} Aug 04 18:49:55.563387 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.563286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" event={"ID":"4682b881-744d-45c2-9ca9-e90ceaa1c654","Type":"ContainerStarted","Data":"303f3957f68cc5ead782f2a815d09f14eeac300a4c54cc25c3898692c490009d"} Aug 04 18:49:55.563654 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.563638 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:49:55.582895 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.582835 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" podStartSLOduration=1.370121447 podStartE2EDuration="2.582816633s" totalImagesPullingTime="1.212695186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:55.582503853 +0000 UTC m=+2360.729320114" watchObservedRunningTime="2026-08-04 18:49:55.582816633 +0000 UTC m=+2360.729632894" Aug 04 18:49:55.620077 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.620021 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" podStartSLOduration=1.620002312 podStartE2EDuration="1.620002312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:55.60216623 +0000 UTC m=+2360.748982490" watchObservedRunningTime="2026-08-04 18:49:55.620002312 +0000 UTC m=+2360.766818574" Aug 04 18:49:55.621668 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:49:55.621628 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" podStartSLOduration=1.621617509 podStartE2EDuration="1.621617509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:55.620150987 +0000 UTC m=+2360.766967224" watchObservedRunningTime="2026-08-04 18:49:55.621617509 +0000 UTC m=+2360.768433769" Aug 04 18:50:01.575572 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:50:01.575542 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:50:06.572440 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:50:06.572409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:50:06.572826 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:50:06.572487 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:50:35.512798 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:50:35.512771 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:50:35.521020 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:50:35.520996 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:51:17.910163 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:17.910129 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh"] Aug 04 18:51:17.910743 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:17.910403 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" podUID="a9402c05-7880-4112-b746-b89ca9a2ad44" containerName="maas-api" containerID="cri-o://9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61" gracePeriod=30 Aug 04 18:51:18.030557 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.030041 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h"] Aug 04 18:51:18.031116 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.031061 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" podUID="4682b881-744d-45c2-9ca9-e90ceaa1c654" containerName="payload-processing" containerID="cri-o://e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa" gracePeriod=30 Aug 04 18:51:18.033694 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.033670 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw"] Aug 04 18:51:18.033881 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.033838 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" podUID="94facd6b-dc41-46f9-9aa5-8720ec291f05" containerName="payload-pre-processing" containerID="cri-o://e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0" gracePeriod=30 Aug 04 18:51:18.193588 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.193560 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:51:18.311821 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.311795 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:51:18.318968 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.318947 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:51:18.338179 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.338151 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjgv2\" (UniqueName: \"kubernetes.io/projected/a9402c05-7880-4112-b746-b89ca9a2ad44-kube-api-access-tjgv2\") pod \"a9402c05-7880-4112-b746-b89ca9a2ad44\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " Aug 04 18:51:18.338309 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.338237 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9402c05-7880-4112-b746-b89ca9a2ad44-maas-api-tls\") pod \"a9402c05-7880-4112-b746-b89ca9a2ad44\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " Aug 04 18:51:18.338399 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.338322 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9402c05-7880-4112-b746-b89ca9a2ad44-openshift-service-ca\") pod \"a9402c05-7880-4112-b746-b89ca9a2ad44\" (UID: \"a9402c05-7880-4112-b746-b89ca9a2ad44\") " Aug 04 18:51:18.338982 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.338944 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9402c05-7880-4112-b746-b89ca9a2ad44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9402c05-7880-4112-b746-b89ca9a2ad44" (UID: "a9402c05-7880-4112-b746-b89ca9a2ad44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:18.341264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.341235 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9402c05-7880-4112-b746-b89ca9a2ad44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9402c05-7880-4112-b746-b89ca9a2ad44" (UID: "a9402c05-7880-4112-b746-b89ca9a2ad44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:51:18.341264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.341246 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9402c05-7880-4112-b746-b89ca9a2ad44-kube-api-access-tjgv2" (OuterVolumeSpecName: "kube-api-access-tjgv2") pod "a9402c05-7880-4112-b746-b89ca9a2ad44" (UID: "a9402c05-7880-4112-b746-b89ca9a2ad44"). InnerVolumeSpecName "kube-api-access-tjgv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:18.439587 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.439511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnbx8\" (UniqueName: \"kubernetes.io/projected/94facd6b-dc41-46f9-9aa5-8720ec291f05-kube-api-access-fnbx8\") pod \"94facd6b-dc41-46f9-9aa5-8720ec291f05\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " Aug 04 18:51:18.439587 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.439563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4682b881-744d-45c2-9ca9-e90ceaa1c654-plugins-config-volume\") pod \"4682b881-744d-45c2-9ca9-e90ceaa1c654\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " Aug 04 18:51:18.439587 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.439579 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94facd6b-dc41-46f9-9aa5-8720ec291f05-plugins-config-volume\") pod \"94facd6b-dc41-46f9-9aa5-8720ec291f05\" (UID: \"94facd6b-dc41-46f9-9aa5-8720ec291f05\") " Aug 04 18:51:18.439791 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.439660 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlk42\" (UniqueName: \"kubernetes.io/projected/4682b881-744d-45c2-9ca9-e90ceaa1c654-kube-api-access-vlk42\") pod \"4682b881-744d-45c2-9ca9-e90ceaa1c654\" (UID: \"4682b881-744d-45c2-9ca9-e90ceaa1c654\") " Aug 04 18:51:18.439918 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.439896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4682b881-744d-45c2-9ca9-e90ceaa1c654-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4682b881-744d-45c2-9ca9-e90ceaa1c654" (UID: "4682b881-744d-45c2-9ca9-e90ceaa1c654"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:18.440015 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.439987 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94facd6b-dc41-46f9-9aa5-8720ec291f05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94facd6b-dc41-46f9-9aa5-8720ec291f05" (UID: "94facd6b-dc41-46f9-9aa5-8720ec291f05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:18.440126 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.440043 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9402c05-7880-4112-b746-b89ca9a2ad44-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:18.440126 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.440063 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9402c05-7880-4112-b746-b89ca9a2ad44-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:18.440126 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.440078 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgv2\" (UniqueName: \"kubernetes.io/projected/a9402c05-7880-4112-b746-b89ca9a2ad44-kube-api-access-tjgv2\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:18.440126 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.440086 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4682b881-744d-45c2-9ca9-e90ceaa1c654-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:18.441865 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.441844 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4682b881-744d-45c2-9ca9-e90ceaa1c654-kube-api-access-vlk42" (OuterVolumeSpecName: "kube-api-access-vlk42") pod "4682b881-744d-45c2-9ca9-e90ceaa1c654" (UID: "4682b881-744d-45c2-9ca9-e90ceaa1c654"). InnerVolumeSpecName "kube-api-access-vlk42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:18.441953 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.441934 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94facd6b-dc41-46f9-9aa5-8720ec291f05-kube-api-access-fnbx8" (OuterVolumeSpecName: "kube-api-access-fnbx8") pod "94facd6b-dc41-46f9-9aa5-8720ec291f05" (UID: "94facd6b-dc41-46f9-9aa5-8720ec291f05"). InnerVolumeSpecName "kube-api-access-fnbx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:18.540752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.540718 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlk42\" (UniqueName: \"kubernetes.io/projected/4682b881-744d-45c2-9ca9-e90ceaa1c654-kube-api-access-vlk42\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:18.540752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.540745 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbx8\" (UniqueName: \"kubernetes.io/projected/94facd6b-dc41-46f9-9aa5-8720ec291f05-kube-api-access-fnbx8\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:18.540752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:18.540755 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94facd6b-dc41-46f9-9aa5-8720ec291f05-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:51:19.004653 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.004619 2575 generic.go:356] "Generic (PLEG): container finished" podID="94facd6b-dc41-46f9-9aa5-8720ec291f05" containerID="e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0" exitCode=0 Aug 04 18:51:19.005037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.004676 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" Aug 04 18:51:19.005037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.004703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" event={"ID":"94facd6b-dc41-46f9-9aa5-8720ec291f05","Type":"ContainerDied","Data":"e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0"} Aug 04 18:51:19.005037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.004768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw" event={"ID":"94facd6b-dc41-46f9-9aa5-8720ec291f05","Type":"ContainerDied","Data":"2ac8c8a1ef57497cdc485f2700bb4254a6225f16a38dca1418057fe19f77f8b3"} Aug 04 18:51:19.005037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.004803 2575 scope.go:117] "RemoveContainer" containerID="e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0" Aug 04 18:51:19.006092 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.006072 2575 generic.go:356] "Generic (PLEG): container finished" podID="4682b881-744d-45c2-9ca9-e90ceaa1c654" containerID="e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa" exitCode=0 Aug 04 18:51:19.006154 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.006121 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" Aug 04 18:51:19.006209 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.006151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" event={"ID":"4682b881-744d-45c2-9ca9-e90ceaa1c654","Type":"ContainerDied","Data":"e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa"} Aug 04 18:51:19.006251 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.006192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h" event={"ID":"4682b881-744d-45c2-9ca9-e90ceaa1c654","Type":"ContainerDied","Data":"303f3957f68cc5ead782f2a815d09f14eeac300a4c54cc25c3898692c490009d"} Aug 04 18:51:19.007707 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.007682 2575 generic.go:356] "Generic (PLEG): container finished" podID="a9402c05-7880-4112-b746-b89ca9a2ad44" containerID="9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61" exitCode=0 Aug 04 18:51:19.007833 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.007710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" event={"ID":"a9402c05-7880-4112-b746-b89ca9a2ad44","Type":"ContainerDied","Data":"9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61"} Aug 04 18:51:19.007833 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.007726 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" Aug 04 18:51:19.007833 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.007765 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh" event={"ID":"a9402c05-7880-4112-b746-b89ca9a2ad44","Type":"ContainerDied","Data":"cb17c3fde3289546e2ffc8c559fa2b27475b0ab9a67a8031f604291ebb661956"} Aug 04 18:51:19.016289 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.016270 2575 scope.go:117] "RemoveContainer" containerID="e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0" Aug 04 18:51:19.016597 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:51:19.016572 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0\": container with ID starting with e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0 not found: ID does not exist" containerID="e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0" Aug 04 18:51:19.016732 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.016606 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0"} err="failed to get container status \"e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0\": rpc error: code = NotFound desc = could not find container \"e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0\": container with ID starting with e5918a67feb0c0ce289d4b7870f12793742b7b5e596f781e8210bbc58a2125b0 not found: ID does not exist" Aug 04 18:51:19.016732 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.016628 2575 scope.go:117] "RemoveContainer" containerID="e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa" Aug 04 18:51:19.025133 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.025118 2575 scope.go:117] "RemoveContainer" containerID="e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa" Aug 04 18:51:19.025383 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:51:19.025355 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa\": container with ID starting with e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa not found: ID does not exist" containerID="e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa" Aug 04 18:51:19.025474 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.025387 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa"} err="failed to get container status \"e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa\": rpc error: code = NotFound desc = could not find container \"e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa\": container with ID starting with e1cd639586a49432b4ccb98c798895e41799e1ca2318b8b1e6b8ce4df20b40fa not found: ID does not exist" Aug 04 18:51:19.025474 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.025401 2575 scope.go:117] "RemoveContainer" containerID="9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61" Aug 04 18:51:19.032851 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.032832 2575 scope.go:117] "RemoveContainer" containerID="9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61" Aug 04 18:51:19.033175 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:51:19.033147 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61\": container with ID starting with 9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61 not found: ID does not exist" containerID="9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61" Aug 04 18:51:19.033312 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.033178 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61"} err="failed to get container status \"9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61\": rpc error: code = NotFound desc = could not find container \"9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61\": container with ID starting with 9d83e4390b023adb7cda366f903088434bddccbbfb9a25578ea3992507e67c61 not found: ID does not exist" Aug 04 18:51:19.039023 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.038998 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw"] Aug 04 18:51:19.043432 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.043411 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee88ca8-6dd8f65d88-rjwcw"] Aug 04 18:51:19.055699 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.055677 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh"] Aug 04 18:51:19.057629 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.057609 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee88ca8-798f476f69-pvszh"] Aug 04 18:51:19.070920 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.070900 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h"] Aug 04 18:51:19.078043 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.078024 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee88ca8-6454888978-mmp4h"] Aug 04 18:51:19.349706 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.349619 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4682b881-744d-45c2-9ca9-e90ceaa1c654" path="/var/lib/kubelet/pods/4682b881-744d-45c2-9ca9-e90ceaa1c654/volumes" Aug 04 18:51:19.350048 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.350027 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94facd6b-dc41-46f9-9aa5-8720ec291f05" path="/var/lib/kubelet/pods/94facd6b-dc41-46f9-9aa5-8720ec291f05/volumes" Aug 04 18:51:19.350450 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:51:19.350432 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9402c05-7880-4112-b746-b89ca9a2ad44" path="/var/lib/kubelet/pods/a9402c05-7880-4112-b746-b89ca9a2ad44/volumes" Aug 04 18:53:23.341388 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341334 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc"] Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341688 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4682b881-744d-45c2-9ca9-e90ceaa1c654" containerName="payload-processing" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341700 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4682b881-744d-45c2-9ca9-e90ceaa1c654" containerName="payload-processing" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341712 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9402c05-7880-4112-b746-b89ca9a2ad44" containerName="maas-api" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341718 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9402c05-7880-4112-b746-b89ca9a2ad44" containerName="maas-api" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341727 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94facd6b-dc41-46f9-9aa5-8720ec291f05" containerName="payload-pre-processing" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341733 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="94facd6b-dc41-46f9-9aa5-8720ec291f05" containerName="payload-pre-processing" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341787 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4682b881-744d-45c2-9ca9-e90ceaa1c654" containerName="payload-processing" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341795 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="94facd6b-dc41-46f9-9aa5-8720ec291f05" containerName="payload-pre-processing" Aug 04 18:53:23.343530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.341800 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9402c05-7880-4112-b746-b89ca9a2ad44" containerName="maas-api" Aug 04 18:53:23.344594 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.344569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.347309 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.347290 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:53:23.347448 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.347333 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:53:23.348237 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.348220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-890d1a-serving-cert\"" Aug 04 18:53:23.348332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.348315 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:53:23.359023 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.359000 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc"] Aug 04 18:53:23.442203 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.442169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeea2589-7872-49de-b2bd-5e7b9a68443e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.442417 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.442251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mbr\" (UniqueName: \"kubernetes.io/projected/eeea2589-7872-49de-b2bd-5e7b9a68443e-kube-api-access-d8mbr\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.442417 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.442285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.543523 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.543485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeea2589-7872-49de-b2bd-5e7b9a68443e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.543719 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.543548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mbr\" (UniqueName: \"kubernetes.io/projected/eeea2589-7872-49de-b2bd-5e7b9a68443e-kube-api-access-d8mbr\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.543719 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.543570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.543719 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:53:23.543686 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-serving-cert: secret "maas-api-e2e-inf-a-890d1a-serving-cert" not found Aug 04 18:53:23.543895 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:53:23.543740 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls podName:eeea2589-7872-49de-b2bd-5e7b9a68443e nodeName:}" failed. No retries permitted until 2026-08-04 18:53:24.043724034 +0000 UTC m=+2569.190540271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls") pod "maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" (UID: "eeea2589-7872-49de-b2bd-5e7b9a68443e") : secret "maas-api-e2e-inf-a-890d1a-serving-cert" not found Aug 04 18:53:23.544137 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.544115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeea2589-7872-49de-b2bd-5e7b9a68443e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:23.561806 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:23.561779 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mbr\" (UniqueName: \"kubernetes.io/projected/eeea2589-7872-49de-b2bd-5e7b9a68443e-kube-api-access-d8mbr\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:24.047762 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.047730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:24.050438 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.050404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls\") pod \"maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:24.241052 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.241009 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf"] Aug 04 18:53:24.244469 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.244445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.247055 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.247032 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-890d1a-dockercfg-j2mdv\"" Aug 04 18:53:24.247157 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.247124 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-890d1a\"" Aug 04 18:53:24.255458 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.255434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:24.255607 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.255584 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf"] Aug 04 18:53:24.349357 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.349307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qxw\" (UniqueName: \"kubernetes.io/projected/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-kube-api-access-r2qxw\") pod \"payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.349768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.349402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.397595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.397509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc"] Aug 04 18:53:24.399827 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:53:24.399801 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeea2589_7872_49de_b2bd_5e7b9a68443e.slice/crio-8787d7ade0d0336d3a672396e01a00a7c09bcca2ff443eb9770470f312ca13ba WatchSource:0}: Error finding container 8787d7ade0d0336d3a672396e01a00a7c09bcca2ff443eb9770470f312ca13ba: Status 404 returned error can't find the container with id 8787d7ade0d0336d3a672396e01a00a7c09bcca2ff443eb9770470f312ca13ba Aug 04 18:53:24.450651 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.450625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qxw\" (UniqueName: \"kubernetes.io/projected/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-kube-api-access-r2qxw\") pod \"payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.450800 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.450686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.451340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.451321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.460232 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.460204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qxw\" (UniqueName: \"kubernetes.io/projected/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-kube-api-access-r2qxw\") pod \"payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.556760 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.556729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:24.672436 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.672399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" event={"ID":"eeea2589-7872-49de-b2bd-5e7b9a68443e","Type":"ContainerStarted","Data":"8787d7ade0d0336d3a672396e01a00a7c09bcca2ff443eb9770470f312ca13ba"} Aug 04 18:53:24.704018 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:24.703987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf"] Aug 04 18:53:24.705381 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:53:24.705338 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b0b4c3_882d_421b_842a_0f7bdbf11f90.slice/crio-42ba494a2982544f3c23d63956e092d972f3531eb3b09190531c49d6920bf584 WatchSource:0}: Error finding container 42ba494a2982544f3c23d63956e092d972f3531eb3b09190531c49d6920bf584: Status 404 returned error can't find the container with id 42ba494a2982544f3c23d63956e092d972f3531eb3b09190531c49d6920bf584 Aug 04 18:53:25.678876 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:25.678832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" event={"ID":"eeea2589-7872-49de-b2bd-5e7b9a68443e","Type":"ContainerStarted","Data":"8559032435c6712a1c9e50ef4cfcf8418bc05ee4579782f4e397a85c6c4bdcd8"} Aug 04 18:53:25.679332 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:25.678971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:25.680420 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:25.680396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" event={"ID":"c9b0b4c3-882d-421b-842a-0f7bdbf11f90","Type":"ContainerStarted","Data":"cfa34389ff73113016e9ce11a761d8e1c9b185d7091848b5fd7c6f8f63aff00a"} Aug 04 18:53:25.680557 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:25.680439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" event={"ID":"c9b0b4c3-882d-421b-842a-0f7bdbf11f90","Type":"ContainerStarted","Data":"42ba494a2982544f3c23d63956e092d972f3531eb3b09190531c49d6920bf584"} Aug 04 18:53:25.680557 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:25.680465 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:25.705173 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:25.705117 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" podStartSLOduration=2.0920009999999998 podStartE2EDuration="2.705099288s" totalImagesPullingTime="613.098288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:25.703883229 +0000 UTC m=+2570.850699489" watchObservedRunningTime="2026-08-04 18:53:25.705099288 +0000 UTC m=+2570.851915546" Aug 04 18:53:31.690969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:31.690939 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:53:31.717116 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:31.717060 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" podStartSLOduration=7.717043995 podStartE2EDuration="7.717043995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:25.728510216 +0000 UTC m=+2570.875326474" watchObservedRunningTime="2026-08-04 18:53:31.717043995 +0000 UTC m=+2576.863860263" Aug 04 18:53:36.687879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:36.687847 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:53:57.483165 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.483079 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4"] Aug 04 18:53:57.486903 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.486883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.491614 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.491595 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-716d7c-serving-cert\"" Aug 04 18:53:57.505557 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.505532 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4"] Aug 04 18:53:57.652689 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.652650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a93ab9-44f5-49ea-84dc-262dab4d8964-openshift-service-ca\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.652689 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.652695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjgk\" (UniqueName: \"kubernetes.io/projected/11a93ab9-44f5-49ea-84dc-262dab4d8964-kube-api-access-qwjgk\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.652899 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.652753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a93ab9-44f5-49ea-84dc-262dab4d8964-maas-api-tls\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.753869 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.753778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjgk\" (UniqueName: \"kubernetes.io/projected/11a93ab9-44f5-49ea-84dc-262dab4d8964-kube-api-access-qwjgk\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.753869 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.753830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a93ab9-44f5-49ea-84dc-262dab4d8964-maas-api-tls\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.754085 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.753912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a93ab9-44f5-49ea-84dc-262dab4d8964-openshift-service-ca\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.754676 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.754650 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a93ab9-44f5-49ea-84dc-262dab4d8964-openshift-service-ca\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.756594 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.756566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a93ab9-44f5-49ea-84dc-262dab4d8964-maas-api-tls\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.765999 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.765973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjgk\" (UniqueName: \"kubernetes.io/projected/11a93ab9-44f5-49ea-84dc-262dab4d8964-kube-api-access-qwjgk\") pod \"maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.797458 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.797422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:57.945263 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:57.945240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4"] Aug 04 18:53:57.947582 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:53:57.947556 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a93ab9_44f5_49ea_84dc_262dab4d8964.slice/crio-ac7634b56675363ed4d925b7e8ffb57f10e14b77fa814dea65a7b777d9f1df3c WatchSource:0}: Error finding container ac7634b56675363ed4d925b7e8ffb57f10e14b77fa814dea65a7b777d9f1df3c: Status 404 returned error can't find the container with id ac7634b56675363ed4d925b7e8ffb57f10e14b77fa814dea65a7b777d9f1df3c Aug 04 18:53:58.305135 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.305096 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz"] Aug 04 18:53:58.309963 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.309940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.312699 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.312679 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-716d7c\"" Aug 04 18:53:58.312817 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.312700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-716d7c-dockercfg-6bpj7\"" Aug 04 18:53:58.321977 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.321956 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz"] Aug 04 18:53:58.382959 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.382924 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw"] Aug 04 18:53:58.386346 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.386329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.396151 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.396126 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw"] Aug 04 18:53:58.459920 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.459886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ww49\" (UniqueName: \"kubernetes.io/projected/cfe9baf6-58a0-41dd-81ef-d705952dd900-kube-api-access-2ww49\") pod \"payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.460101 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.460018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe9baf6-58a0-41dd-81ef-d705952dd900-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.561407 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.561376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe9baf6-58a0-41dd-81ef-d705952dd900-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.561739 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.561494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b5e64-e7cb-45a3-bcf7-bd3987586e67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.561739 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.561547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ww49\" (UniqueName: \"kubernetes.io/projected/cfe9baf6-58a0-41dd-81ef-d705952dd900-kube-api-access-2ww49\") pod \"payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.561739 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.561583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlq2\" (UniqueName: \"kubernetes.io/projected/796b5e64-e7cb-45a3-bcf7-bd3987586e67-kube-api-access-smlq2\") pod \"payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.562118 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.562096 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe9baf6-58a0-41dd-81ef-d705952dd900-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.574551 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.574527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ww49\" (UniqueName: \"kubernetes.io/projected/cfe9baf6-58a0-41dd-81ef-d705952dd900-kube-api-access-2ww49\") pod \"payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.621513 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.621473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:58.663061 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.662964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b5e64-e7cb-45a3-bcf7-bd3987586e67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.663061 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.663039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlq2\" (UniqueName: \"kubernetes.io/projected/796b5e64-e7cb-45a3-bcf7-bd3987586e67-kube-api-access-smlq2\") pod \"payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.663882 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.663856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b5e64-e7cb-45a3-bcf7-bd3987586e67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.677115 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.677049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlq2\" (UniqueName: \"kubernetes.io/projected/796b5e64-e7cb-45a3-bcf7-bd3987586e67-kube-api-access-smlq2\") pod \"payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.698466 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.698430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:58.775595 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.775557 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz"] Aug 04 18:53:58.778079 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:53:58.778046 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe9baf6_58a0_41dd_81ef_d705952dd900.slice/crio-d815e58e77ad6ccdb5fdfe910b9502fc991d2d2bc0f850f5c4954cf850960457 WatchSource:0}: Error finding container d815e58e77ad6ccdb5fdfe910b9502fc991d2d2bc0f850f5c4954cf850960457: Status 404 returned error can't find the container with id d815e58e77ad6ccdb5fdfe910b9502fc991d2d2bc0f850f5c4954cf850960457 Aug 04 18:53:58.863771 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.863743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw"] Aug 04 18:53:58.865819 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:53:58.865789 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796b5e64_e7cb_45a3_bcf7_bd3987586e67.slice/crio-59c277476b62d5d96eefc4ed0b8657060ee90bade71c3745b14dd9c6ffabf7bf WatchSource:0}: Error finding container 59c277476b62d5d96eefc4ed0b8657060ee90bade71c3745b14dd9c6ffabf7bf: Status 404 returned error can't find the container with id 59c277476b62d5d96eefc4ed0b8657060ee90bade71c3745b14dd9c6ffabf7bf Aug 04 18:53:58.869534 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.869498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" event={"ID":"cfe9baf6-58a0-41dd-81ef-d705952dd900","Type":"ContainerStarted","Data":"d815e58e77ad6ccdb5fdfe910b9502fc991d2d2bc0f850f5c4954cf850960457"} Aug 04 18:53:58.871488 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.871460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" event={"ID":"11a93ab9-44f5-49ea-84dc-262dab4d8964","Type":"ContainerStarted","Data":"8f069b8df5c2bce5ca5514f11374761b0f5a2e9e390205a3b1a0cfc88df68c95"} Aug 04 18:53:58.871597 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.871515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" event={"ID":"11a93ab9-44f5-49ea-84dc-262dab4d8964","Type":"ContainerStarted","Data":"ac7634b56675363ed4d925b7e8ffb57f10e14b77fa814dea65a7b777d9f1df3c"} Aug 04 18:53:58.873346 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.873130 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:53:58.894404 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:58.894325 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" podStartSLOduration=1.313054851 podStartE2EDuration="1.89430289s" totalImagesPullingTime="581.248039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:58.89299886 +0000 UTC m=+2604.039815122" watchObservedRunningTime="2026-08-04 18:53:58.89430289 +0000 UTC m=+2604.041119153" Aug 04 18:53:59.878881 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.878828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" event={"ID":"796b5e64-e7cb-45a3-bcf7-bd3987586e67","Type":"ContainerStarted","Data":"4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02"} Aug 04 18:53:59.879391 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.878895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" event={"ID":"796b5e64-e7cb-45a3-bcf7-bd3987586e67","Type":"ContainerStarted","Data":"59c277476b62d5d96eefc4ed0b8657060ee90bade71c3745b14dd9c6ffabf7bf"} Aug 04 18:53:59.879391 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.879038 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:53:59.880659 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.880634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" event={"ID":"cfe9baf6-58a0-41dd-81ef-d705952dd900","Type":"ContainerStarted","Data":"6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f"} Aug 04 18:53:59.880786 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.880755 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:53:59.910136 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.910053 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" podStartSLOduration=1.9100375299999999 podStartE2EDuration="1.91003753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:59.908167085 +0000 UTC m=+2605.054983346" watchObservedRunningTime="2026-08-04 18:53:59.91003753 +0000 UTC m=+2605.056853790" Aug 04 18:53:59.933437 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:53:59.933355 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" podStartSLOduration=1.9333366650000001 podStartE2EDuration="1.933336665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:59.932947934 +0000 UTC m=+2605.079764195" watchObservedRunningTime="2026-08-04 18:53:59.933336665 +0000 UTC m=+2605.080152925" Aug 04 18:54:04.886186 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:54:04.886156 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:54:10.889392 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:54:10.889332 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:54:10.889768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:54:10.889545 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:55:35.549094 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:35.549011 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:55:35.559300 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:35.559273 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 18:55:47.195525 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.195489 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc"] Aug 04 18:55:47.195977 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.195688 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" podUID="eeea2589-7872-49de-b2bd-5e7b9a68443e" containerName="maas-api" containerID="cri-o://8559032435c6712a1c9e50ef4cfcf8418bc05ee4579782f4e397a85c6c4bdcd8" gracePeriod=30 Aug 04 18:55:47.341356 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.341315 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf"] Aug 04 18:55:47.341635 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.341585 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" podUID="c9b0b4c3-882d-421b-842a-0f7bdbf11f90" containerName="payload-pre-processing" containerID="cri-o://cfa34389ff73113016e9ce11a761d8e1c9b185d7091848b5fd7c6f8f63aff00a" gracePeriod=30 Aug 04 18:55:47.460718 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.460565 2575 generic.go:356] "Generic (PLEG): container finished" podID="c9b0b4c3-882d-421b-842a-0f7bdbf11f90" containerID="cfa34389ff73113016e9ce11a761d8e1c9b185d7091848b5fd7c6f8f63aff00a" exitCode=0 Aug 04 18:55:47.460718 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.460638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" event={"ID":"c9b0b4c3-882d-421b-842a-0f7bdbf11f90","Type":"ContainerDied","Data":"cfa34389ff73113016e9ce11a761d8e1c9b185d7091848b5fd7c6f8f63aff00a"} Aug 04 18:55:47.462681 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.462655 2575 generic.go:356] "Generic (PLEG): container finished" podID="eeea2589-7872-49de-b2bd-5e7b9a68443e" containerID="8559032435c6712a1c9e50ef4cfcf8418bc05ee4579782f4e397a85c6c4bdcd8" exitCode=0 Aug 04 18:55:47.462786 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.462730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" event={"ID":"eeea2589-7872-49de-b2bd-5e7b9a68443e","Type":"ContainerDied","Data":"8559032435c6712a1c9e50ef4cfcf8418bc05ee4579782f4e397a85c6c4bdcd8"} Aug 04 18:55:47.478877 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.478846 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:55:47.549170 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.549126 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls\") pod \"eeea2589-7872-49de-b2bd-5e7b9a68443e\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " Aug 04 18:55:47.549339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.549209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeea2589-7872-49de-b2bd-5e7b9a68443e-openshift-service-ca\") pod \"eeea2589-7872-49de-b2bd-5e7b9a68443e\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " Aug 04 18:55:47.549339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.549267 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8mbr\" (UniqueName: \"kubernetes.io/projected/eeea2589-7872-49de-b2bd-5e7b9a68443e-kube-api-access-d8mbr\") pod \"eeea2589-7872-49de-b2bd-5e7b9a68443e\" (UID: \"eeea2589-7872-49de-b2bd-5e7b9a68443e\") " Aug 04 18:55:47.550100 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.550069 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeea2589-7872-49de-b2bd-5e7b9a68443e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eeea2589-7872-49de-b2bd-5e7b9a68443e" (UID: "eeea2589-7872-49de-b2bd-5e7b9a68443e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:55:47.553261 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.553214 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeea2589-7872-49de-b2bd-5e7b9a68443e-kube-api-access-d8mbr" (OuterVolumeSpecName: "kube-api-access-d8mbr") pod "eeea2589-7872-49de-b2bd-5e7b9a68443e" (UID: "eeea2589-7872-49de-b2bd-5e7b9a68443e"). InnerVolumeSpecName "kube-api-access-d8mbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:55:47.553976 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.553934 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eeea2589-7872-49de-b2bd-5e7b9a68443e" (UID: "eeea2589-7872-49de-b2bd-5e7b9a68443e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:55:47.600666 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.600636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:55:47.650234 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.650204 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-plugins-config-volume\") pod \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " Aug 04 18:55:47.650531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.650242 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2qxw\" (UniqueName: \"kubernetes.io/projected/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-kube-api-access-r2qxw\") pod \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\" (UID: \"c9b0b4c3-882d-421b-842a-0f7bdbf11f90\") " Aug 04 18:55:47.650613 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.650527 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeea2589-7872-49de-b2bd-5e7b9a68443e-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:55:47.650613 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.650547 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mbr\" (UniqueName: \"kubernetes.io/projected/eeea2589-7872-49de-b2bd-5e7b9a68443e-kube-api-access-d8mbr\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:55:47.650613 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.650563 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeea2589-7872-49de-b2bd-5e7b9a68443e-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:55:47.650764 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.650677 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9b0b4c3-882d-421b-842a-0f7bdbf11f90" (UID: "c9b0b4c3-882d-421b-842a-0f7bdbf11f90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:55:47.652700 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.652674 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-kube-api-access-r2qxw" (OuterVolumeSpecName: "kube-api-access-r2qxw") pod "c9b0b4c3-882d-421b-842a-0f7bdbf11f90" (UID: "c9b0b4c3-882d-421b-842a-0f7bdbf11f90"). InnerVolumeSpecName "kube-api-access-r2qxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:55:47.751320 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.751235 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:55:47.751320 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:47.751263 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qxw\" (UniqueName: \"kubernetes.io/projected/c9b0b4c3-882d-421b-842a-0f7bdbf11f90-kube-api-access-r2qxw\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:55:48.470317 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.470221 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" Aug 04 18:55:48.470317 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.470248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc" event={"ID":"eeea2589-7872-49de-b2bd-5e7b9a68443e","Type":"ContainerDied","Data":"8787d7ade0d0336d3a672396e01a00a7c09bcca2ff443eb9770470f312ca13ba"} Aug 04 18:55:48.470317 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.470306 2575 scope.go:117] "RemoveContainer" containerID="8559032435c6712a1c9e50ef4cfcf8418bc05ee4579782f4e397a85c6c4bdcd8" Aug 04 18:55:48.471762 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.471738 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" event={"ID":"c9b0b4c3-882d-421b-842a-0f7bdbf11f90","Type":"ContainerDied","Data":"42ba494a2982544f3c23d63956e092d972f3531eb3b09190531c49d6920bf584"} Aug 04 18:55:48.471836 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.471766 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf" Aug 04 18:55:48.485227 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.485202 2575 scope.go:117] "RemoveContainer" containerID="cfa34389ff73113016e9ce11a761d8e1c9b185d7091848b5fd7c6f8f63aff00a" Aug 04 18:55:48.499468 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.499438 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc"] Aug 04 18:55:48.507801 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.507770 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-890d1a-7b5f7cd5fd-h2xxc"] Aug 04 18:55:48.519694 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.519664 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf"] Aug 04 18:55:48.527316 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:48.527283 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-890d1a-8f86bdcfb-b6ltf"] Aug 04 18:55:49.354319 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:49.354283 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b0b4c3-882d-421b-842a-0f7bdbf11f90" path="/var/lib/kubelet/pods/c9b0b4c3-882d-421b-842a-0f7bdbf11f90/volumes" Aug 04 18:55:49.354700 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:55:49.354686 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeea2589-7872-49de-b2bd-5e7b9a68443e" path="/var/lib/kubelet/pods/eeea2589-7872-49de-b2bd-5e7b9a68443e/volumes" Aug 04 18:56:11.161789 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.161739 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4"] Aug 04 18:56:11.162378 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.162029 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" podUID="11a93ab9-44f5-49ea-84dc-262dab4d8964" containerName="maas-api" containerID="cri-o://8f069b8df5c2bce5ca5514f11374761b0f5a2e9e390205a3b1a0cfc88df68c95" gracePeriod=30 Aug 04 18:56:11.237988 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.237938 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw"] Aug 04 18:56:11.238228 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.238203 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" podUID="796b5e64-e7cb-45a3-bcf7-bd3987586e67" containerName="payload-processing" containerID="cri-o://4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02" gracePeriod=30 Aug 04 18:56:11.244506 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.244472 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz"] Aug 04 18:56:11.244722 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.244694 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" podUID="cfe9baf6-58a0-41dd-81ef-d705952dd900" containerName="payload-pre-processing" containerID="cri-o://6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f" gracePeriod=30 Aug 04 18:56:11.514135 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.514107 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:56:11.520497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.520474 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:56:11.597830 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.597799 2575 generic.go:356] "Generic (PLEG): container finished" podID="796b5e64-e7cb-45a3-bcf7-bd3987586e67" containerID="4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02" exitCode=0 Aug 04 18:56:11.598000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.597833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" event={"ID":"796b5e64-e7cb-45a3-bcf7-bd3987586e67","Type":"ContainerDied","Data":"4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02"} Aug 04 18:56:11.598000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.597897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" event={"ID":"796b5e64-e7cb-45a3-bcf7-bd3987586e67","Type":"ContainerDied","Data":"59c277476b62d5d96eefc4ed0b8657060ee90bade71c3745b14dd9c6ffabf7bf"} Aug 04 18:56:11.598000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.597880 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw" Aug 04 18:56:11.598000 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.597921 2575 scope.go:117] "RemoveContainer" containerID="4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02" Aug 04 18:56:11.599256 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.599235 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfe9baf6-58a0-41dd-81ef-d705952dd900" containerID="6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f" exitCode=0 Aug 04 18:56:11.599383 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.599267 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" Aug 04 18:56:11.599383 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.599326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" event={"ID":"cfe9baf6-58a0-41dd-81ef-d705952dd900","Type":"ContainerDied","Data":"6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f"} Aug 04 18:56:11.599494 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.599383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz" event={"ID":"cfe9baf6-58a0-41dd-81ef-d705952dd900","Type":"ContainerDied","Data":"d815e58e77ad6ccdb5fdfe910b9502fc991d2d2bc0f850f5c4954cf850960457"} Aug 04 18:56:11.600915 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.600897 2575 generic.go:356] "Generic (PLEG): container finished" podID="11a93ab9-44f5-49ea-84dc-262dab4d8964" containerID="8f069b8df5c2bce5ca5514f11374761b0f5a2e9e390205a3b1a0cfc88df68c95" exitCode=0 Aug 04 18:56:11.600996 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.600970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" event={"ID":"11a93ab9-44f5-49ea-84dc-262dab4d8964","Type":"ContainerDied","Data":"8f069b8df5c2bce5ca5514f11374761b0f5a2e9e390205a3b1a0cfc88df68c95"} Aug 04 18:56:11.607591 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.607578 2575 scope.go:117] "RemoveContainer" containerID="4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02" Aug 04 18:56:11.607864 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:56:11.607844 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02\": container with ID starting with 4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02 not found: ID does not exist" containerID="4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02" Aug 04 18:56:11.607916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.607867 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02"} err="failed to get container status \"4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02\": rpc error: code = NotFound desc = could not find container \"4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02\": container with ID starting with 4d8e901e4bbc5dd4733e1c43619c3826521f816e2ee8c13e979f3138c612fb02 not found: ID does not exist" Aug 04 18:56:11.607916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.607881 2575 scope.go:117] "RemoveContainer" containerID="6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f" Aug 04 18:56:11.618209 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.618188 2575 scope.go:117] "RemoveContainer" containerID="6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f" Aug 04 18:56:11.618549 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:56:11.618532 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f\": container with ID starting with 6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f not found: ID does not exist" containerID="6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f" Aug 04 18:56:11.618697 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.618555 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f"} err="failed to get container status \"6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f\": rpc error: code = NotFound desc = could not find container \"6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f\": container with ID starting with 6b9620218b7b4c315e6257b11f465d00da2dc4fe41441102537a85ad4afd758f not found: ID does not exist" Aug 04 18:56:11.635916 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.635893 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:56:11.676155 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.676127 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe9baf6-58a0-41dd-81ef-d705952dd900-plugins-config-volume\") pod \"cfe9baf6-58a0-41dd-81ef-d705952dd900\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " Aug 04 18:56:11.676325 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.676182 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ww49\" (UniqueName: \"kubernetes.io/projected/cfe9baf6-58a0-41dd-81ef-d705952dd900-kube-api-access-2ww49\") pod \"cfe9baf6-58a0-41dd-81ef-d705952dd900\" (UID: \"cfe9baf6-58a0-41dd-81ef-d705952dd900\") " Aug 04 18:56:11.676325 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.676216 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smlq2\" (UniqueName: \"kubernetes.io/projected/796b5e64-e7cb-45a3-bcf7-bd3987586e67-kube-api-access-smlq2\") pod \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " Aug 04 18:56:11.676325 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.676251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b5e64-e7cb-45a3-bcf7-bd3987586e67-plugins-config-volume\") pod \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\" (UID: \"796b5e64-e7cb-45a3-bcf7-bd3987586e67\") " Aug 04 18:56:11.676659 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.676635 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe9baf6-58a0-41dd-81ef-d705952dd900-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfe9baf6-58a0-41dd-81ef-d705952dd900" (UID: "cfe9baf6-58a0-41dd-81ef-d705952dd900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:56:11.676723 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.676700 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796b5e64-e7cb-45a3-bcf7-bd3987586e67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "796b5e64-e7cb-45a3-bcf7-bd3987586e67" (UID: "796b5e64-e7cb-45a3-bcf7-bd3987586e67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:56:11.678646 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.678623 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe9baf6-58a0-41dd-81ef-d705952dd900-kube-api-access-2ww49" (OuterVolumeSpecName: "kube-api-access-2ww49") pod "cfe9baf6-58a0-41dd-81ef-d705952dd900" (UID: "cfe9baf6-58a0-41dd-81ef-d705952dd900"). InnerVolumeSpecName "kube-api-access-2ww49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:11.678761 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.678648 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796b5e64-e7cb-45a3-bcf7-bd3987586e67-kube-api-access-smlq2" (OuterVolumeSpecName: "kube-api-access-smlq2") pod "796b5e64-e7cb-45a3-bcf7-bd3987586e67" (UID: "796b5e64-e7cb-45a3-bcf7-bd3987586e67"). InnerVolumeSpecName "kube-api-access-smlq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:11.777187 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777093 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a93ab9-44f5-49ea-84dc-262dab4d8964-openshift-service-ca\") pod \"11a93ab9-44f5-49ea-84dc-262dab4d8964\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " Aug 04 18:56:11.777187 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a93ab9-44f5-49ea-84dc-262dab4d8964-maas-api-tls\") pod \"11a93ab9-44f5-49ea-84dc-262dab4d8964\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " Aug 04 18:56:11.777187 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777181 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwjgk\" (UniqueName: \"kubernetes.io/projected/11a93ab9-44f5-49ea-84dc-262dab4d8964-kube-api-access-qwjgk\") pod \"11a93ab9-44f5-49ea-84dc-262dab4d8964\" (UID: \"11a93ab9-44f5-49ea-84dc-262dab4d8964\") " Aug 04 18:56:11.777497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777479 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe9baf6-58a0-41dd-81ef-d705952dd900-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.777549 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777504 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ww49\" (UniqueName: \"kubernetes.io/projected/cfe9baf6-58a0-41dd-81ef-d705952dd900-kube-api-access-2ww49\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.777549 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777508 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a93ab9-44f5-49ea-84dc-262dab4d8964-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11a93ab9-44f5-49ea-84dc-262dab4d8964" (UID: "11a93ab9-44f5-49ea-84dc-262dab4d8964"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:56:11.777549 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777519 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smlq2\" (UniqueName: \"kubernetes.io/projected/796b5e64-e7cb-45a3-bcf7-bd3987586e67-kube-api-access-smlq2\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.777661 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.777553 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796b5e64-e7cb-45a3-bcf7-bd3987586e67-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.779491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.779459 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a93ab9-44f5-49ea-84dc-262dab4d8964-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11a93ab9-44f5-49ea-84dc-262dab4d8964" (UID: "11a93ab9-44f5-49ea-84dc-262dab4d8964"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:56:11.779619 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.779578 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a93ab9-44f5-49ea-84dc-262dab4d8964-kube-api-access-qwjgk" (OuterVolumeSpecName: "kube-api-access-qwjgk") pod "11a93ab9-44f5-49ea-84dc-262dab4d8964" (UID: "11a93ab9-44f5-49ea-84dc-262dab4d8964"). InnerVolumeSpecName "kube-api-access-qwjgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:56:11.878473 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.878430 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a93ab9-44f5-49ea-84dc-262dab4d8964-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.878473 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.878465 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a93ab9-44f5-49ea-84dc-262dab4d8964-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.878473 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.878475 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwjgk\" (UniqueName: \"kubernetes.io/projected/11a93ab9-44f5-49ea-84dc-262dab4d8964-kube-api-access-qwjgk\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:56:11.938011 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.937976 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz"] Aug 04 18:56:11.941897 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.941865 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-716d7c-78bd465564-g4qnz"] Aug 04 18:56:11.955691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.955661 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw"] Aug 04 18:56:11.967096 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:11.967063 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-716d7c-78b5f96bc6-clfbw"] Aug 04 18:56:12.609184 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:12.609100 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" Aug 04 18:56:12.609184 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:12.609117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4" event={"ID":"11a93ab9-44f5-49ea-84dc-262dab4d8964","Type":"ContainerDied","Data":"ac7634b56675363ed4d925b7e8ffb57f10e14b77fa814dea65a7b777d9f1df3c"} Aug 04 18:56:12.609184 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:12.609178 2575 scope.go:117] "RemoveContainer" containerID="8f069b8df5c2bce5ca5514f11374761b0f5a2e9e390205a3b1a0cfc88df68c95" Aug 04 18:56:12.632665 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:12.632634 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4"] Aug 04 18:56:12.637223 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:12.637194 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-716d7c-9dd866fb4-6c7q4"] Aug 04 18:56:13.348944 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:13.348910 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11a93ab9-44f5-49ea-84dc-262dab4d8964" path="/var/lib/kubelet/pods/11a93ab9-44f5-49ea-84dc-262dab4d8964/volumes" Aug 04 18:56:13.349264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:13.349251 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796b5e64-e7cb-45a3-bcf7-bd3987586e67" path="/var/lib/kubelet/pods/796b5e64-e7cb-45a3-bcf7-bd3987586e67/volumes" Aug 04 18:56:13.349596 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:13.349584 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe9baf6-58a0-41dd-81ef-d705952dd900" path="/var/lib/kubelet/pods/cfe9baf6-58a0-41dd-81ef-d705952dd900/volumes" Aug 04 18:56:24.211659 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.211626 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v"] Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.211983 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a93ab9-44f5-49ea-84dc-262dab4d8964" containerName="maas-api" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.211993 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a93ab9-44f5-49ea-84dc-262dab4d8964" containerName="maas-api" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212011 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b0b4c3-882d-421b-842a-0f7bdbf11f90" containerName="payload-pre-processing" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212017 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b0b4c3-882d-421b-842a-0f7bdbf11f90" containerName="payload-pre-processing" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212026 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeea2589-7872-49de-b2bd-5e7b9a68443e" containerName="maas-api" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212031 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeea2589-7872-49de-b2bd-5e7b9a68443e" containerName="maas-api" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212038 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796b5e64-e7cb-45a3-bcf7-bd3987586e67" containerName="payload-processing" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212044 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="796b5e64-e7cb-45a3-bcf7-bd3987586e67" containerName="payload-processing" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212059 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe9baf6-58a0-41dd-81ef-d705952dd900" containerName="payload-pre-processing" Aug 04 18:56:24.212062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212064 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe9baf6-58a0-41dd-81ef-d705952dd900" containerName="payload-pre-processing" Aug 04 18:56:24.212497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212116 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="796b5e64-e7cb-45a3-bcf7-bd3987586e67" containerName="payload-processing" Aug 04 18:56:24.212497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212124 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="11a93ab9-44f5-49ea-84dc-262dab4d8964" containerName="maas-api" Aug 04 18:56:24.212497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212132 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeea2589-7872-49de-b2bd-5e7b9a68443e" containerName="maas-api" Aug 04 18:56:24.212497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212138 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe9baf6-58a0-41dd-81ef-d705952dd900" containerName="payload-pre-processing" Aug 04 18:56:24.212497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.212144 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b0b4c3-882d-421b-842a-0f7bdbf11f90" containerName="payload-pre-processing" Aug 04 18:56:24.216335 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.216315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.219056 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.219032 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:56:24.219174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.219031 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-95504a-serving-cert\"" Aug 04 18:56:24.219876 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.219862 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:56:24.219918 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.219879 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:56:24.227589 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.227564 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v"] Aug 04 18:56:24.287232 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.287200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd29308-d723-4c4d-969a-19a16272966d-openshift-service-ca\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.287437 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.287241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.287437 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.287277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpkq\" (UniqueName: \"kubernetes.io/projected/8fd29308-d723-4c4d-969a-19a16272966d-kube-api-access-jxpkq\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.388738 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.388703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd29308-d723-4c4d-969a-19a16272966d-openshift-service-ca\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.388903 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.388751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.388903 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.388798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpkq\" (UniqueName: \"kubernetes.io/projected/8fd29308-d723-4c4d-969a-19a16272966d-kube-api-access-jxpkq\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.389016 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:56:24.388919 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-serving-cert: secret "maas-api-e2e-api-a-95504a-serving-cert" not found Aug 04 18:56:24.389016 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:56:24.388996 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls podName:8fd29308-d723-4c4d-969a-19a16272966d nodeName:}" failed. No retries permitted until 2026-08-04 18:56:24.888979241 +0000 UTC m=+2750.035795478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls") pod "maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" (UID: "8fd29308-d723-4c4d-969a-19a16272966d") : secret "maas-api-e2e-api-a-95504a-serving-cert" not found Aug 04 18:56:24.389331 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.389314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd29308-d723-4c4d-969a-19a16272966d-openshift-service-ca\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.398019 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.397995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpkq\" (UniqueName: \"kubernetes.io/projected/8fd29308-d723-4c4d-969a-19a16272966d-kube-api-access-jxpkq\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.892793 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.892756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:24.895372 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:24.895342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls\") pod \"maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:25.006255 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.006215 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm"] Aug 04 18:56:25.009907 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.009889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.012827 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.012802 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-95504a\"" Aug 04 18:56:25.012949 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.012855 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-95504a-dockercfg-2rkx6\"" Aug 04 18:56:25.021815 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.021789 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm"] Aug 04 18:56:25.078648 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.078607 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl"] Aug 04 18:56:25.082053 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.082033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.093927 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.093900 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl"] Aug 04 18:56:25.094484 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.094459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcldb\" (UniqueName: \"kubernetes.io/projected/28315995-b85c-4399-a5f7-2c65c36ee7a8-kube-api-access-kcldb\") pod \"payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.094540 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.094514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3abf33-a268-40d1-b46a-6349f8dd331d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.094665 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.094645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkp7\" (UniqueName: \"kubernetes.io/projected/7f3abf33-a268-40d1-b46a-6349f8dd331d-kube-api-access-ddkp7\") pod \"payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.094734 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.094688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28315995-b85c-4399-a5f7-2c65c36ee7a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.129244 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.129215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:25.196230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.195270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3abf33-a268-40d1-b46a-6349f8dd331d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.196230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.195408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkp7\" (UniqueName: \"kubernetes.io/projected/7f3abf33-a268-40d1-b46a-6349f8dd331d-kube-api-access-ddkp7\") pod \"payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.196230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.195467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28315995-b85c-4399-a5f7-2c65c36ee7a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.196230 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.195538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcldb\" (UniqueName: \"kubernetes.io/projected/28315995-b85c-4399-a5f7-2c65c36ee7a8-kube-api-access-kcldb\") pod \"payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.197153 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.196948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3abf33-a268-40d1-b46a-6349f8dd331d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.197153 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.196969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28315995-b85c-4399-a5f7-2c65c36ee7a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.205079 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.205010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkp7\" (UniqueName: \"kubernetes.io/projected/7f3abf33-a268-40d1-b46a-6349f8dd331d-kube-api-access-ddkp7\") pod \"payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.205196 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.205148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcldb\" (UniqueName: \"kubernetes.io/projected/28315995-b85c-4399-a5f7-2c65c36ee7a8-kube-api-access-kcldb\") pod \"payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.272903 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.272854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v"] Aug 04 18:56:25.275160 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:56:25.275130 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd29308_d723_4c4d_969a_19a16272966d.slice/crio-058e92581f439f613ef87cc0b04f99f868f33eefa150bc65e1c352bc194f97b0 WatchSource:0}: Error finding container 058e92581f439f613ef87cc0b04f99f868f33eefa150bc65e1c352bc194f97b0: Status 404 returned error can't find the container with id 058e92581f439f613ef87cc0b04f99f868f33eefa150bc65e1c352bc194f97b0 Aug 04 18:56:25.276472 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.276454 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:56:25.321435 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.321401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.393901 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.393869 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.484472 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.484443 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm"] Aug 04 18:56:25.486482 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:56:25.486447 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28315995_b85c_4399_a5f7_2c65c36ee7a8.slice/crio-f52641854a94d086ae13dbd5e01d33e1207ca76eca51b94b1eb0668ea5342a96 WatchSource:0}: Error finding container f52641854a94d086ae13dbd5e01d33e1207ca76eca51b94b1eb0668ea5342a96: Status 404 returned error can't find the container with id f52641854a94d086ae13dbd5e01d33e1207ca76eca51b94b1eb0668ea5342a96 Aug 04 18:56:25.555083 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.555057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl"] Aug 04 18:56:25.556724 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:56:25.556693 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3abf33_a268_40d1_b46a_6349f8dd331d.slice/crio-b1d9cfa33938473a4e4a958c049a51dac90fc414a7a2039408921feccd277f8e WatchSource:0}: Error finding container b1d9cfa33938473a4e4a958c049a51dac90fc414a7a2039408921feccd277f8e: Status 404 returned error can't find the container with id b1d9cfa33938473a4e4a958c049a51dac90fc414a7a2039408921feccd277f8e Aug 04 18:56:25.694120 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.694068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" event={"ID":"28315995-b85c-4399-a5f7-2c65c36ee7a8","Type":"ContainerStarted","Data":"dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295"} Aug 04 18:56:25.694322 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.694163 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" event={"ID":"28315995-b85c-4399-a5f7-2c65c36ee7a8","Type":"ContainerStarted","Data":"f52641854a94d086ae13dbd5e01d33e1207ca76eca51b94b1eb0668ea5342a96"} Aug 04 18:56:25.694322 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.694282 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:25.696211 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.696180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" event={"ID":"7f3abf33-a268-40d1-b46a-6349f8dd331d","Type":"ContainerStarted","Data":"3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a"} Aug 04 18:56:25.696404 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.696234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" event={"ID":"7f3abf33-a268-40d1-b46a-6349f8dd331d","Type":"ContainerStarted","Data":"b1d9cfa33938473a4e4a958c049a51dac90fc414a7a2039408921feccd277f8e"} Aug 04 18:56:25.696404 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.696263 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:25.697923 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.697892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" event={"ID":"8fd29308-d723-4c4d-969a-19a16272966d","Type":"ContainerStarted","Data":"058e92581f439f613ef87cc0b04f99f868f33eefa150bc65e1c352bc194f97b0"} Aug 04 18:56:25.717929 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.717815 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" podStartSLOduration=1.71779722 podStartE2EDuration="1.71779722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:25.715802621 +0000 UTC m=+2750.862618892" watchObservedRunningTime="2026-08-04 18:56:25.71779722 +0000 UTC m=+2750.864613522" Aug 04 18:56:25.740043 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:25.739978 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" podStartSLOduration=0.739960184 podStartE2EDuration="739.960184ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:25.738712 +0000 UTC m=+2750.885528261" watchObservedRunningTime="2026-08-04 18:56:25.739960184 +0000 UTC m=+2750.886776443" Aug 04 18:56:26.707716 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:26.707661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" event={"ID":"8fd29308-d723-4c4d-969a-19a16272966d","Type":"ContainerStarted","Data":"e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46"} Aug 04 18:56:26.708136 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:26.708089 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:26.728536 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:26.728480 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" podStartSLOduration=1.632464938 podStartE2EDuration="2.728462873s" totalImagesPullingTime="1.095997935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:26.727136387 +0000 UTC m=+2751.873952650" watchObservedRunningTime="2026-08-04 18:56:26.728462873 +0000 UTC m=+2751.875279131" Aug 04 18:56:33.724135 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:33.724100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:56:36.709745 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:36.709691 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" podUID="28315995-b85c-4399-a5f7-2c65c36ee7a8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.67:9004: connect: connection refused" Aug 04 18:56:36.710169 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:36.709970 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:56:46.708879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:46.708841 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:56:53.536442 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.536397 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2"] Aug 04 18:56:53.540520 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.540499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.544167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.544147 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0de7a2-serving-cert\"" Aug 04 18:56:53.554619 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.554588 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2"] Aug 04 18:56:53.666094 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.666050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4xx\" (UniqueName: \"kubernetes.io/projected/45e83d08-c692-4a93-bb9f-5eb6bd748321-kube-api-access-sj4xx\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.666302 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.666107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45e83d08-c692-4a93-bb9f-5eb6bd748321-maas-api-tls\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.666302 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.666134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e83d08-c692-4a93-bb9f-5eb6bd748321-openshift-service-ca\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.767612 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.767575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4xx\" (UniqueName: \"kubernetes.io/projected/45e83d08-c692-4a93-bb9f-5eb6bd748321-kube-api-access-sj4xx\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.767830 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.767644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45e83d08-c692-4a93-bb9f-5eb6bd748321-maas-api-tls\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.767830 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.767687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e83d08-c692-4a93-bb9f-5eb6bd748321-openshift-service-ca\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.768504 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.768483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e83d08-c692-4a93-bb9f-5eb6bd748321-openshift-service-ca\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.770222 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.770188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45e83d08-c692-4a93-bb9f-5eb6bd748321-maas-api-tls\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.777339 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.777312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4xx\" (UniqueName: \"kubernetes.io/projected/45e83d08-c692-4a93-bb9f-5eb6bd748321-kube-api-access-sj4xx\") pod \"maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.851103 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.851004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:53.993239 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:53.993215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2"] Aug 04 18:56:53.995130 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:56:53.995102 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e83d08_c692_4a93_bb9f_5eb6bd748321.slice/crio-36e42dc9dc359c108efedeeed9db744f02101be5f30cf03967421fb806798406 WatchSource:0}: Error finding container 36e42dc9dc359c108efedeeed9db744f02101be5f30cf03967421fb806798406: Status 404 returned error can't find the container with id 36e42dc9dc359c108efedeeed9db744f02101be5f30cf03967421fb806798406 Aug 04 18:56:54.243380 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.243318 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24"] Aug 04 18:56:54.248061 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.248038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.251530 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.251508 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0de7a2\"" Aug 04 18:56:54.251768 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.251752 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0de7a2-dockercfg-8z8gv\"" Aug 04 18:56:54.259160 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.259131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24"] Aug 04 18:56:54.289291 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.289262 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6"] Aug 04 18:56:54.292939 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.292914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.304891 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.304862 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6"] Aug 04 18:56:54.374283 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.374241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqb59\" (UniqueName: \"kubernetes.io/projected/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-kube-api-access-tqb59\") pod \"payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.374474 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.374395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.475574 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.475535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqb59\" (UniqueName: \"kubernetes.io/projected/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-kube-api-access-tqb59\") pod \"payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.475574 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.475582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ng8\" (UniqueName: \"kubernetes.io/projected/3b648073-99c5-425f-83aa-7a7ae77b5295-kube-api-access-76ng8\") pod \"payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.475848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.475640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b648073-99c5-425f-83aa-7a7ae77b5295-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.475848 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.475660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.476427 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.476405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.489720 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.489693 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqb59\" (UniqueName: \"kubernetes.io/projected/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-kube-api-access-tqb59\") pod \"payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.560576 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.560487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.576990 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.576957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ng8\" (UniqueName: \"kubernetes.io/projected/3b648073-99c5-425f-83aa-7a7ae77b5295-kube-api-access-76ng8\") pod \"payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.577121 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.577036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b648073-99c5-425f-83aa-7a7ae77b5295-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.577730 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.577711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b648073-99c5-425f-83aa-7a7ae77b5295-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.586756 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.586732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ng8\" (UniqueName: \"kubernetes.io/projected/3b648073-99c5-425f-83aa-7a7ae77b5295-kube-api-access-76ng8\") pod \"payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.605216 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.605178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:54.716981 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.716948 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24"] Aug 04 18:56:54.721346 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:56:54.721311 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c915fe5_ab4b_4fd5_8e93_0fb9552f2c1b.slice/crio-1ec8d280636775cef6b95c4e52a62c2bf58c534c781b0f78e80d32c4a0b325af WatchSource:0}: Error finding container 1ec8d280636775cef6b95c4e52a62c2bf58c534c781b0f78e80d32c4a0b325af: Status 404 returned error can't find the container with id 1ec8d280636775cef6b95c4e52a62c2bf58c534c781b0f78e80d32c4a0b325af Aug 04 18:56:54.767388 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.767312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6"] Aug 04 18:56:54.774575 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:56:54.774547 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b648073_99c5_425f_83aa_7a7ae77b5295.slice/crio-e784cd04fb24919687473584d7bb2b5973c73807e9620d52f3d21f2ae7692348 WatchSource:0}: Error finding container e784cd04fb24919687473584d7bb2b5973c73807e9620d52f3d21f2ae7692348: Status 404 returned error can't find the container with id e784cd04fb24919687473584d7bb2b5973c73807e9620d52f3d21f2ae7692348 Aug 04 18:56:54.864662 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.864630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" event={"ID":"3b648073-99c5-425f-83aa-7a7ae77b5295","Type":"ContainerStarted","Data":"e784cd04fb24919687473584d7bb2b5973c73807e9620d52f3d21f2ae7692348"} Aug 04 18:56:54.866606 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.866574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" event={"ID":"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b","Type":"ContainerStarted","Data":"c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469"} Aug 04 18:56:54.866751 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.866633 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" event={"ID":"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b","Type":"ContainerStarted","Data":"1ec8d280636775cef6b95c4e52a62c2bf58c534c781b0f78e80d32c4a0b325af"} Aug 04 18:56:54.866751 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.866673 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:56:54.868433 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.868406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" event={"ID":"45e83d08-c692-4a93-bb9f-5eb6bd748321","Type":"ContainerStarted","Data":"20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7"} Aug 04 18:56:54.868554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.868458 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:56:54.868554 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.868483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" event={"ID":"45e83d08-c692-4a93-bb9f-5eb6bd748321","Type":"ContainerStarted","Data":"36e42dc9dc359c108efedeeed9db744f02101be5f30cf03967421fb806798406"} Aug 04 18:56:54.891909 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.891733 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" podStartSLOduration=0.89171329 podStartE2EDuration="891.71329ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:54.891068522 +0000 UTC m=+2780.037884783" watchObservedRunningTime="2026-08-04 18:56:54.89171329 +0000 UTC m=+2780.038529552" Aug 04 18:56:54.914445 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:54.914394 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" podStartSLOduration=1.6117045970000001 podStartE2EDuration="1.914344911s" totalImagesPullingTime="302.640314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:54.913472186 +0000 UTC m=+2780.060288448" watchObservedRunningTime="2026-08-04 18:56:54.914344911 +0000 UTC m=+2780.061161170" Aug 04 18:56:55.875176 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:55.875077 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" event={"ID":"3b648073-99c5-425f-83aa-7a7ae77b5295","Type":"ContainerStarted","Data":"31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299"} Aug 04 18:56:55.875668 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:55.875171 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:56:55.895376 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:56:55.895312 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" podStartSLOduration=1.895295184 podStartE2EDuration="1.895295184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:55.892974184 +0000 UTC m=+2781.039790443" watchObservedRunningTime="2026-08-04 18:56:55.895295184 +0000 UTC m=+2781.042111445" Aug 04 18:57:00.881302 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:00.881271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:57:05.878182 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:05.878154 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:57:06.884436 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:06.884404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:57:36.250811 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.250651 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v"] Aug 04 18:57:36.251505 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.250977 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" podUID="8fd29308-d723-4c4d-969a-19a16272966d" containerName="maas-api" containerID="cri-o://e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46" gracePeriod=30 Aug 04 18:57:36.317232 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.317197 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl"] Aug 04 18:57:36.317474 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.317444 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" podUID="7f3abf33-a268-40d1-b46a-6349f8dd331d" containerName="payload-processing" containerID="cri-o://3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a" gracePeriod=30 Aug 04 18:57:36.325782 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.325724 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm"] Aug 04 18:57:36.326093 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.325942 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" podUID="28315995-b85c-4399-a5f7-2c65c36ee7a8" containerName="payload-pre-processing" containerID="cri-o://dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295" gracePeriod=30 Aug 04 18:57:36.605012 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.604982 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:57:36.617450 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.617421 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:57:36.630337 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.630307 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:57:36.778457 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778339 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls\") pod \"8fd29308-d723-4c4d-969a-19a16272966d\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " Aug 04 18:57:36.778457 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778410 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd29308-d723-4c4d-969a-19a16272966d-openshift-service-ca\") pod \"8fd29308-d723-4c4d-969a-19a16272966d\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " Aug 04 18:57:36.778457 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778456 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28315995-b85c-4399-a5f7-2c65c36ee7a8-plugins-config-volume\") pod \"28315995-b85c-4399-a5f7-2c65c36ee7a8\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " Aug 04 18:57:36.778773 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778496 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3abf33-a268-40d1-b46a-6349f8dd331d-plugins-config-volume\") pod \"7f3abf33-a268-40d1-b46a-6349f8dd331d\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " Aug 04 18:57:36.778773 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778563 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcldb\" (UniqueName: \"kubernetes.io/projected/28315995-b85c-4399-a5f7-2c65c36ee7a8-kube-api-access-kcldb\") pod \"28315995-b85c-4399-a5f7-2c65c36ee7a8\" (UID: \"28315995-b85c-4399-a5f7-2c65c36ee7a8\") " Aug 04 18:57:36.778773 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778621 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxpkq\" (UniqueName: \"kubernetes.io/projected/8fd29308-d723-4c4d-969a-19a16272966d-kube-api-access-jxpkq\") pod \"8fd29308-d723-4c4d-969a-19a16272966d\" (UID: \"8fd29308-d723-4c4d-969a-19a16272966d\") " Aug 04 18:57:36.778773 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778659 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddkp7\" (UniqueName: \"kubernetes.io/projected/7f3abf33-a268-40d1-b46a-6349f8dd331d-kube-api-access-ddkp7\") pod \"7f3abf33-a268-40d1-b46a-6349f8dd331d\" (UID: \"7f3abf33-a268-40d1-b46a-6349f8dd331d\") " Aug 04 18:57:36.778991 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778827 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd29308-d723-4c4d-969a-19a16272966d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fd29308-d723-4c4d-969a-19a16272966d" (UID: "8fd29308-d723-4c4d-969a-19a16272966d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:36.778991 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778851 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28315995-b85c-4399-a5f7-2c65c36ee7a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28315995-b85c-4399-a5f7-2c65c36ee7a8" (UID: "28315995-b85c-4399-a5f7-2c65c36ee7a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:36.778991 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.778896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3abf33-a268-40d1-b46a-6349f8dd331d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f3abf33-a268-40d1-b46a-6349f8dd331d" (UID: "7f3abf33-a268-40d1-b46a-6349f8dd331d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:36.779144 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.779004 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd29308-d723-4c4d-969a-19a16272966d-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:36.779144 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.779022 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28315995-b85c-4399-a5f7-2c65c36ee7a8-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:36.779144 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.779039 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3abf33-a268-40d1-b46a-6349f8dd331d-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:36.780988 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.780957 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fd29308-d723-4c4d-969a-19a16272966d" (UID: "8fd29308-d723-4c4d-969a-19a16272966d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:57:36.780988 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.780958 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3abf33-a268-40d1-b46a-6349f8dd331d-kube-api-access-ddkp7" (OuterVolumeSpecName: "kube-api-access-ddkp7") pod "7f3abf33-a268-40d1-b46a-6349f8dd331d" (UID: "7f3abf33-a268-40d1-b46a-6349f8dd331d"). InnerVolumeSpecName "kube-api-access-ddkp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:36.781245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.781221 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28315995-b85c-4399-a5f7-2c65c36ee7a8-kube-api-access-kcldb" (OuterVolumeSpecName: "kube-api-access-kcldb") pod "28315995-b85c-4399-a5f7-2c65c36ee7a8" (UID: "28315995-b85c-4399-a5f7-2c65c36ee7a8"). InnerVolumeSpecName "kube-api-access-kcldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:36.781459 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.781436 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd29308-d723-4c4d-969a-19a16272966d-kube-api-access-jxpkq" (OuterVolumeSpecName: "kube-api-access-jxpkq") pod "8fd29308-d723-4c4d-969a-19a16272966d" (UID: "8fd29308-d723-4c4d-969a-19a16272966d"). InnerVolumeSpecName "kube-api-access-jxpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:36.880029 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.879977 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxpkq\" (UniqueName: \"kubernetes.io/projected/8fd29308-d723-4c4d-969a-19a16272966d-kube-api-access-jxpkq\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:36.880029 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.880024 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddkp7\" (UniqueName: \"kubernetes.io/projected/7f3abf33-a268-40d1-b46a-6349f8dd331d-kube-api-access-ddkp7\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:36.880029 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.880039 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd29308-d723-4c4d-969a-19a16272966d-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:36.880296 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:36.880053 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcldb\" (UniqueName: \"kubernetes.io/projected/28315995-b85c-4399-a5f7-2c65c36ee7a8-kube-api-access-kcldb\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:37.117022 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.116932 2575 generic.go:356] "Generic (PLEG): container finished" podID="28315995-b85c-4399-a5f7-2c65c36ee7a8" containerID="dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295" exitCode=0 Aug 04 18:57:37.117022 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.116978 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" Aug 04 18:57:37.117274 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.117018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" event={"ID":"28315995-b85c-4399-a5f7-2c65c36ee7a8","Type":"ContainerDied","Data":"dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295"} Aug 04 18:57:37.117274 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.117071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm" event={"ID":"28315995-b85c-4399-a5f7-2c65c36ee7a8","Type":"ContainerDied","Data":"f52641854a94d086ae13dbd5e01d33e1207ca76eca51b94b1eb0668ea5342a96"} Aug 04 18:57:37.117274 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.117093 2575 scope.go:117] "RemoveContainer" containerID="dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295" Aug 04 18:57:37.118403 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.118381 2575 generic.go:356] "Generic (PLEG): container finished" podID="7f3abf33-a268-40d1-b46a-6349f8dd331d" containerID="3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a" exitCode=0 Aug 04 18:57:37.118531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.118432 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" Aug 04 18:57:37.118531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.118467 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" event={"ID":"7f3abf33-a268-40d1-b46a-6349f8dd331d","Type":"ContainerDied","Data":"3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a"} Aug 04 18:57:37.118531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.118506 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl" event={"ID":"7f3abf33-a268-40d1-b46a-6349f8dd331d","Type":"ContainerDied","Data":"b1d9cfa33938473a4e4a958c049a51dac90fc414a7a2039408921feccd277f8e"} Aug 04 18:57:37.120184 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.120152 2575 generic.go:356] "Generic (PLEG): container finished" podID="8fd29308-d723-4c4d-969a-19a16272966d" containerID="e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46" exitCode=0 Aug 04 18:57:37.120283 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.120179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" event={"ID":"8fd29308-d723-4c4d-969a-19a16272966d","Type":"ContainerDied","Data":"e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46"} Aug 04 18:57:37.120283 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.120161 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" Aug 04 18:57:37.120283 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.120227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v" event={"ID":"8fd29308-d723-4c4d-969a-19a16272966d","Type":"ContainerDied","Data":"058e92581f439f613ef87cc0b04f99f868f33eefa150bc65e1c352bc194f97b0"} Aug 04 18:57:37.129205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.129186 2575 scope.go:117] "RemoveContainer" containerID="dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295" Aug 04 18:57:37.129633 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:57:37.129597 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295\": container with ID starting with dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295 not found: ID does not exist" containerID="dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295" Aug 04 18:57:37.129741 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.129644 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295"} err="failed to get container status \"dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295\": rpc error: code = NotFound desc = could not find container \"dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295\": container with ID starting with dc5d1e0098fec27c7624f83cb2045a4a40f03dbe78d06dc8fb44e03a7c683295 not found: ID does not exist" Aug 04 18:57:37.129741 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.129665 2575 scope.go:117] "RemoveContainer" containerID="3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a" Aug 04 18:57:37.138685 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.138667 2575 scope.go:117] "RemoveContainer" containerID="3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a" Aug 04 18:57:37.138957 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:57:37.138932 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a\": container with ID starting with 3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a not found: ID does not exist" containerID="3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a" Aug 04 18:57:37.139014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.138967 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a"} err="failed to get container status \"3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a\": rpc error: code = NotFound desc = could not find container \"3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a\": container with ID starting with 3a0d6a4a7413350385951b974e3fee2d1cc4aadfb349298811a09c312fdf3d3a not found: ID does not exist" Aug 04 18:57:37.139014 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.138990 2575 scope.go:117] "RemoveContainer" containerID="e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46" Aug 04 18:57:37.144676 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.144654 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm"] Aug 04 18:57:37.148255 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.148232 2575 scope.go:117] "RemoveContainer" containerID="e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46" Aug 04 18:57:37.148568 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:57:37.148551 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46\": container with ID starting with e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46 not found: ID does not exist" containerID="e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46" Aug 04 18:57:37.148627 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.148574 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46"} err="failed to get container status \"e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46\": rpc error: code = NotFound desc = could not find container \"e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46\": container with ID starting with e04e75e53761361cac1700a80101d7f0f3ba92f86773ebabfc81b22c1194aa46 not found: ID does not exist" Aug 04 18:57:37.149585 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.149565 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95504a-67c4bcc9c6-gg6xm"] Aug 04 18:57:37.162252 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.162229 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v"] Aug 04 18:57:37.164700 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.164680 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95504a-5dbc94449d-kvh4v"] Aug 04 18:57:37.176282 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.176253 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl"] Aug 04 18:57:37.180312 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.180290 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95504a-8c74b6bd-2xktl"] Aug 04 18:57:37.349300 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.349255 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28315995-b85c-4399-a5f7-2c65c36ee7a8" path="/var/lib/kubelet/pods/28315995-b85c-4399-a5f7-2c65c36ee7a8/volumes" Aug 04 18:57:37.349854 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.349832 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3abf33-a268-40d1-b46a-6349f8dd331d" path="/var/lib/kubelet/pods/7f3abf33-a268-40d1-b46a-6349f8dd331d/volumes" Aug 04 18:57:37.350274 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:37.350255 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd29308-d723-4c4d-969a-19a16272966d" path="/var/lib/kubelet/pods/8fd29308-d723-4c4d-969a-19a16272966d/volumes" Aug 04 18:57:53.754691 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:53.754643 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2"] Aug 04 18:57:53.755275 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:53.754888 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" podUID="45e83d08-c692-4a93-bb9f-5eb6bd748321" containerName="maas-api" containerID="cri-o://20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7" gracePeriod=30 Aug 04 18:57:53.826173 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:53.826134 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24"] Aug 04 18:57:53.827245 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:53.827083 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" podUID="2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" containerName="payload-processing" containerID="cri-o://c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469" gracePeriod=30 Aug 04 18:57:53.833037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:53.833007 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6"] Aug 04 18:57:53.833293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:53.833244 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" podUID="3b648073-99c5-425f-83aa-7a7ae77b5295" containerName="payload-pre-processing" containerID="cri-o://31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299" gracePeriod=30 Aug 04 18:57:54.125192 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.125164 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:57:54.145631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.145600 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:57:54.158591 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.158567 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:57:54.215969 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.215942 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b648073-99c5-425f-83aa-7a7ae77b5295" containerID="31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299" exitCode=0 Aug 04 18:57:54.216167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.216007 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" Aug 04 18:57:54.216167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.216027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" event={"ID":"3b648073-99c5-425f-83aa-7a7ae77b5295","Type":"ContainerDied","Data":"31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299"} Aug 04 18:57:54.216167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.216091 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6" event={"ID":"3b648073-99c5-425f-83aa-7a7ae77b5295","Type":"ContainerDied","Data":"e784cd04fb24919687473584d7bb2b5973c73807e9620d52f3d21f2ae7692348"} Aug 04 18:57:54.216167 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.216123 2575 scope.go:117] "RemoveContainer" containerID="31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299" Aug 04 18:57:54.217451 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.217432 2575 generic.go:356] "Generic (PLEG): container finished" podID="2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" containerID="c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469" exitCode=0 Aug 04 18:57:54.217558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.217486 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" Aug 04 18:57:54.217558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.217486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" event={"ID":"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b","Type":"ContainerDied","Data":"c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469"} Aug 04 18:57:54.217558 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.217544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24" event={"ID":"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b","Type":"ContainerDied","Data":"1ec8d280636775cef6b95c4e52a62c2bf58c534c781b0f78e80d32c4a0b325af"} Aug 04 18:57:54.218961 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.218939 2575 generic.go:356] "Generic (PLEG): container finished" podID="45e83d08-c692-4a93-bb9f-5eb6bd748321" containerID="20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7" exitCode=0 Aug 04 18:57:54.219070 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.218986 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" Aug 04 18:57:54.219070 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.219006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" event={"ID":"45e83d08-c692-4a93-bb9f-5eb6bd748321","Type":"ContainerDied","Data":"20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7"} Aug 04 18:57:54.219070 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.219044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2" event={"ID":"45e83d08-c692-4a93-bb9f-5eb6bd748321","Type":"ContainerDied","Data":"36e42dc9dc359c108efedeeed9db744f02101be5f30cf03967421fb806798406"} Aug 04 18:57:54.225923 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.225906 2575 scope.go:117] "RemoveContainer" containerID="31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299" Aug 04 18:57:54.226198 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:57:54.226170 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299\": container with ID starting with 31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299 not found: ID does not exist" containerID="31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299" Aug 04 18:57:54.226258 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.226217 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299"} err="failed to get container status \"31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299\": rpc error: code = NotFound desc = could not find container \"31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299\": container with ID starting with 31f53d3c4e5abd322dc27818ca6dd2b3c862fb9f28128a73b47f87b91fbac299 not found: ID does not exist" Aug 04 18:57:54.226258 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.226244 2575 scope.go:117] "RemoveContainer" containerID="c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469" Aug 04 18:57:54.234847 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.234826 2575 scope.go:117] "RemoveContainer" containerID="c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469" Aug 04 18:57:54.235146 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:57:54.235120 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469\": container with ID starting with c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469 not found: ID does not exist" containerID="c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469" Aug 04 18:57:54.235205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.235145 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469"} err="failed to get container status \"c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469\": rpc error: code = NotFound desc = could not find container \"c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469\": container with ID starting with c390ee79a53cb753f6de9203dc5ee7b389c902bb76354e10800008de6d860469 not found: ID does not exist" Aug 04 18:57:54.235205 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.235160 2575 scope.go:117] "RemoveContainer" containerID="20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7" Aug 04 18:57:54.241537 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241508 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b648073-99c5-425f-83aa-7a7ae77b5295-plugins-config-volume\") pod \"3b648073-99c5-425f-83aa-7a7ae77b5295\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " Aug 04 18:57:54.241637 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241600 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqb59\" (UniqueName: \"kubernetes.io/projected/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-kube-api-access-tqb59\") pod \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " Aug 04 18:57:54.241699 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241667 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45e83d08-c692-4a93-bb9f-5eb6bd748321-maas-api-tls\") pod \"45e83d08-c692-4a93-bb9f-5eb6bd748321\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " Aug 04 18:57:54.241752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241703 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj4xx\" (UniqueName: \"kubernetes.io/projected/45e83d08-c692-4a93-bb9f-5eb6bd748321-kube-api-access-sj4xx\") pod \"45e83d08-c692-4a93-bb9f-5eb6bd748321\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " Aug 04 18:57:54.241752 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241737 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e83d08-c692-4a93-bb9f-5eb6bd748321-openshift-service-ca\") pod \"45e83d08-c692-4a93-bb9f-5eb6bd748321\" (UID: \"45e83d08-c692-4a93-bb9f-5eb6bd748321\") " Aug 04 18:57:54.241857 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241770 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76ng8\" (UniqueName: \"kubernetes.io/projected/3b648073-99c5-425f-83aa-7a7ae77b5295-kube-api-access-76ng8\") pod \"3b648073-99c5-425f-83aa-7a7ae77b5295\" (UID: \"3b648073-99c5-425f-83aa-7a7ae77b5295\") " Aug 04 18:57:54.241857 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.241802 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-plugins-config-volume\") pod \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\" (UID: \"2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b\") " Aug 04 18:57:54.242411 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.242351 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" (UID: "2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:54.242652 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.242627 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b648073-99c5-425f-83aa-7a7ae77b5295-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b648073-99c5-425f-83aa-7a7ae77b5295" (UID: "3b648073-99c5-425f-83aa-7a7ae77b5295"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:54.242805 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.242687 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e83d08-c692-4a93-bb9f-5eb6bd748321-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45e83d08-c692-4a93-bb9f-5eb6bd748321" (UID: "45e83d08-c692-4a93-bb9f-5eb6bd748321"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:57:54.244204 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.244166 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e83d08-c692-4a93-bb9f-5eb6bd748321-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45e83d08-c692-4a93-bb9f-5eb6bd748321" (UID: "45e83d08-c692-4a93-bb9f-5eb6bd748321"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:57:54.244204 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.244187 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-kube-api-access-tqb59" (OuterVolumeSpecName: "kube-api-access-tqb59") pod "2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" (UID: "2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b"). InnerVolumeSpecName "kube-api-access-tqb59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:54.244452 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.244327 2575 scope.go:117] "RemoveContainer" containerID="20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7" Aug 04 18:57:54.244546 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.244523 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e83d08-c692-4a93-bb9f-5eb6bd748321-kube-api-access-sj4xx" (OuterVolumeSpecName: "kube-api-access-sj4xx") pod "45e83d08-c692-4a93-bb9f-5eb6bd748321" (UID: "45e83d08-c692-4a93-bb9f-5eb6bd748321"). InnerVolumeSpecName "kube-api-access-sj4xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:54.244704 ip-10-0-142-222 kubenswrapper[2575]: E0804 18:57:54.244658 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7\": container with ID starting with 20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7 not found: ID does not exist" containerID="20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7" Aug 04 18:57:54.244750 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.244712 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7"} err="failed to get container status \"20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7\": rpc error: code = NotFound desc = could not find container \"20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7\": container with ID starting with 20dd865ded04450cfd38637f3c3a7c5ddc1425bc8608d74ef7cf9afe09752bf7 not found: ID does not exist" Aug 04 18:57:54.244884 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.244862 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b648073-99c5-425f-83aa-7a7ae77b5295-kube-api-access-76ng8" (OuterVolumeSpecName: "kube-api-access-76ng8") pod "3b648073-99c5-425f-83aa-7a7ae77b5295" (UID: "3b648073-99c5-425f-83aa-7a7ae77b5295"). InnerVolumeSpecName "kube-api-access-76ng8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343560 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45e83d08-c692-4a93-bb9f-5eb6bd748321-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343589 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj4xx\" (UniqueName: \"kubernetes.io/projected/45e83d08-c692-4a93-bb9f-5eb6bd748321-kube-api-access-sj4xx\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343600 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e83d08-c692-4a93-bb9f-5eb6bd748321-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343609 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ng8\" (UniqueName: \"kubernetes.io/projected/3b648073-99c5-425f-83aa-7a7ae77b5295-kube-api-access-76ng8\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343620 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343633 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b648073-99c5-425f-83aa-7a7ae77b5295-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.343644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.343642 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqb59\" (UniqueName: \"kubernetes.io/projected/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b-kube-api-access-tqb59\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 18:57:54.545902 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.545864 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6"] Aug 04 18:57:54.552264 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.552237 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0de7a2-6b76ccc488-npqp6"] Aug 04 18:57:54.565790 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.565758 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2"] Aug 04 18:57:54.572857 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.572829 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0de7a2-d49d6c554-bb4c2"] Aug 04 18:57:54.585497 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.585471 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24"] Aug 04 18:57:54.589631 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:54.589606 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0de7a2-74bdd4c448-9qh24"] Aug 04 18:57:55.349324 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:55.349289 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" path="/var/lib/kubelet/pods/2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b/volumes" Aug 04 18:57:55.349732 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:55.349690 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b648073-99c5-425f-83aa-7a7ae77b5295" path="/var/lib/kubelet/pods/3b648073-99c5-425f-83aa-7a7ae77b5295/volumes" Aug 04 18:57:55.350021 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:57:55.350008 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e83d08-c692-4a93-bb9f-5eb6bd748321" path="/var/lib/kubelet/pods/45e83d08-c692-4a93-bb9f-5eb6bd748321/volumes" Aug 04 18:58:05.703518 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703485 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8"] Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703818 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28315995-b85c-4399-a5f7-2c65c36ee7a8" containerName="payload-pre-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703829 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="28315995-b85c-4399-a5f7-2c65c36ee7a8" containerName="payload-pre-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703839 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b648073-99c5-425f-83aa-7a7ae77b5295" containerName="payload-pre-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703846 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b648073-99c5-425f-83aa-7a7ae77b5295" containerName="payload-pre-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703860 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e83d08-c692-4a93-bb9f-5eb6bd748321" containerName="maas-api" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703866 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e83d08-c692-4a93-bb9f-5eb6bd748321" containerName="maas-api" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703872 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" containerName="payload-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703877 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" containerName="payload-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703888 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3abf33-a268-40d1-b46a-6349f8dd331d" containerName="payload-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703893 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3abf33-a268-40d1-b46a-6349f8dd331d" containerName="payload-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703899 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd29308-d723-4c4d-969a-19a16272966d" containerName="maas-api" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703904 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd29308-d723-4c4d-969a-19a16272966d" containerName="maas-api" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703960 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e83d08-c692-4a93-bb9f-5eb6bd748321" containerName="maas-api" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703967 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd29308-d723-4c4d-969a-19a16272966d" containerName="maas-api" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703973 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="28315995-b85c-4399-a5f7-2c65c36ee7a8" containerName="payload-pre-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703980 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c915fe5-ab4b-4fd5-8e93-0fb9552f2c1b" containerName="payload-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703987 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3abf33-a268-40d1-b46a-6349f8dd331d" containerName="payload-processing" Aug 04 18:58:05.703985 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.703992 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b648073-99c5-425f-83aa-7a7ae77b5295" containerName="payload-pre-processing" Aug 04 18:58:05.708899 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.708879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.711516 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.711489 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-219d48-serving-cert\"" Aug 04 18:58:05.712350 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.712324 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:58:05.712350 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.712334 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bh448\"" Aug 04 18:58:05.712560 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.712404 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:58:05.718463 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.718427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8"] Aug 04 18:58:05.847286 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.847250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ff10c0-2667-47c7-abef-09728d4ece78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.847465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.847339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ff10c0-2667-47c7-abef-09728d4ece78-maas-api-tls\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.847465 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.847397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzl94\" (UniqueName: \"kubernetes.io/projected/06ff10c0-2667-47c7-abef-09728d4ece78-kube-api-access-zzl94\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.948300 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.948262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ff10c0-2667-47c7-abef-09728d4ece78-maas-api-tls\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.948524 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.948312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzl94\" (UniqueName: \"kubernetes.io/projected/06ff10c0-2667-47c7-abef-09728d4ece78-kube-api-access-zzl94\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.948524 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.948408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ff10c0-2667-47c7-abef-09728d4ece78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.949037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.949009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ff10c0-2667-47c7-abef-09728d4ece78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.950873 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.950852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ff10c0-2667-47c7-abef-09728d4ece78-maas-api-tls\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:05.959087 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:05.959036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzl94\" (UniqueName: \"kubernetes.io/projected/06ff10c0-2667-47c7-abef-09728d4ece78-kube-api-access-zzl94\") pod \"maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:06.021010 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.020984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:06.213257 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.213233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8"] Aug 04 18:58:06.215060 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:58:06.215031 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ff10c0_2667_47c7_abef_09728d4ece78.slice/crio-dc9386276a4727580d456436e269b4f528cfe369e4b54717cbb8196261b87f25 WatchSource:0}: Error finding container dc9386276a4727580d456436e269b4f528cfe369e4b54717cbb8196261b87f25: Status 404 returned error can't find the container with id dc9386276a4727580d456436e269b4f528cfe369e4b54717cbb8196261b87f25 Aug 04 18:58:06.289800 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.289761 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" event={"ID":"06ff10c0-2667-47c7-abef-09728d4ece78","Type":"ContainerStarted","Data":"dc9386276a4727580d456436e269b4f528cfe369e4b54717cbb8196261b87f25"} Aug 04 18:58:06.771541 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.771503 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g"] Aug 04 18:58:06.776122 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.776098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:06.779062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.779042 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-219d48-dockercfg-d96wh\"" Aug 04 18:58:06.779174 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.779042 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-219d48\"" Aug 04 18:58:06.793575 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.793551 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g"] Aug 04 18:58:06.821927 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.821900 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54"] Aug 04 18:58:06.825426 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.825406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:06.836865 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.836843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54"] Aug 04 18:58:06.857375 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.857334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wkr\" (UniqueName: \"kubernetes.io/projected/34710ed6-65be-4d7c-8b50-2c4d3d581f83-kube-api-access-25wkr\") pod \"payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:06.857508 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.857389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34710ed6-65be-4d7c-8b50-2c4d3d581f83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:06.958740 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.958699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105a19a6-4014-48ec-8343-b705bc7b5dbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:06.958912 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.958752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wkr\" (UniqueName: \"kubernetes.io/projected/34710ed6-65be-4d7c-8b50-2c4d3d581f83-kube-api-access-25wkr\") pod \"payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:06.958912 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.958780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34710ed6-65be-4d7c-8b50-2c4d3d581f83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:06.958912 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.958853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznzk\" (UniqueName: \"kubernetes.io/projected/105a19a6-4014-48ec-8343-b705bc7b5dbe-kube-api-access-qznzk\") pod \"payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:06.959531 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.959507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34710ed6-65be-4d7c-8b50-2c4d3d581f83-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:06.968327 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:06.968301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wkr\" (UniqueName: \"kubernetes.io/projected/34710ed6-65be-4d7c-8b50-2c4d3d581f83-kube-api-access-25wkr\") pod \"payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:07.060333 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.060229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105a19a6-4014-48ec-8343-b705bc7b5dbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:07.060539 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.060390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznzk\" (UniqueName: \"kubernetes.io/projected/105a19a6-4014-48ec-8343-b705bc7b5dbe-kube-api-access-qznzk\") pod \"payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:07.060970 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.060949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105a19a6-4014-48ec-8343-b705bc7b5dbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:07.069293 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.069271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznzk\" (UniqueName: \"kubernetes.io/projected/105a19a6-4014-48ec-8343-b705bc7b5dbe-kube-api-access-qznzk\") pod \"payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:07.087183 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.087154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:07.135918 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.135880 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:07.241647 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.241612 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g"] Aug 04 18:58:07.242998 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:58:07.242958 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34710ed6_65be_4d7c_8b50_2c4d3d581f83.slice/crio-6eee7965e8ba0aa1873c5260c6838eb0799a8aeb8505c7eb054f43d1ddbd789e WatchSource:0}: Error finding container 6eee7965e8ba0aa1873c5260c6838eb0799a8aeb8505c7eb054f43d1ddbd789e: Status 404 returned error can't find the container with id 6eee7965e8ba0aa1873c5260c6838eb0799a8aeb8505c7eb054f43d1ddbd789e Aug 04 18:58:07.294649 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.294624 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54"] Aug 04 18:58:07.298610 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:07.298586 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" event={"ID":"34710ed6-65be-4d7c-8b50-2c4d3d581f83","Type":"ContainerStarted","Data":"6eee7965e8ba0aa1873c5260c6838eb0799a8aeb8505c7eb054f43d1ddbd789e"} Aug 04 18:58:07.298730 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:58:07.298682 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105a19a6_4014_48ec_8343_b705bc7b5dbe.slice/crio-6ba7683dcb5111e1478b92f1cd9b43b272ff5b4b6abe9c7fd324c7f1e8bfdf27 WatchSource:0}: Error finding container 6ba7683dcb5111e1478b92f1cd9b43b272ff5b4b6abe9c7fd324c7f1e8bfdf27: Status 404 returned error can't find the container with id 6ba7683dcb5111e1478b92f1cd9b43b272ff5b4b6abe9c7fd324c7f1e8bfdf27 Aug 04 18:58:08.307340 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.307301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" event={"ID":"105a19a6-4014-48ec-8343-b705bc7b5dbe","Type":"ContainerStarted","Data":"ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943"} Aug 04 18:58:08.307787 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.307384 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" event={"ID":"105a19a6-4014-48ec-8343-b705bc7b5dbe","Type":"ContainerStarted","Data":"6ba7683dcb5111e1478b92f1cd9b43b272ff5b4b6abe9c7fd324c7f1e8bfdf27"} Aug 04 18:58:08.307787 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.307417 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:08.308992 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.308961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" event={"ID":"34710ed6-65be-4d7c-8b50-2c4d3d581f83","Type":"ContainerStarted","Data":"a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a"} Aug 04 18:58:08.309122 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.309063 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:08.326125 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.326075 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" podStartSLOduration=2.326058611 podStartE2EDuration="2.326058611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:08.324317493 +0000 UTC m=+2853.471133753" watchObservedRunningTime="2026-08-04 18:58:08.326058611 +0000 UTC m=+2853.472874872" Aug 04 18:58:08.343165 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:08.343112 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" podStartSLOduration=2.343100156 podStartE2EDuration="2.343100156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:08.342699963 +0000 UTC m=+2853.489516223" watchObservedRunningTime="2026-08-04 18:58:08.343100156 +0000 UTC m=+2853.489916416" Aug 04 18:58:09.315579 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:09.315544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" event={"ID":"06ff10c0-2667-47c7-abef-09728d4ece78","Type":"ContainerStarted","Data":"d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf"} Aug 04 18:58:09.316077 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:09.315793 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:09.335271 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:09.335221 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" podStartSLOduration=2.08479562 podStartE2EDuration="4.335205888s" totalImagesPullingTime="2.250410268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:09.332244811 +0000 UTC m=+2854.479061072" watchObservedRunningTime="2026-08-04 18:58:09.335205888 +0000 UTC m=+2854.482022148" Aug 04 18:58:15.326672 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:15.326642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 18:58:19.317559 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:19.317529 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 18:58:19.318041 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:19.317853 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 18:58:40.023906 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.023860 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr"] Aug 04 18:58:40.029085 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.029055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.031547 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.031522 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-de2394-serving-cert\"" Aug 04 18:58:40.038817 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.038758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr"] Aug 04 18:58:40.167250 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.167209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.167250 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.167252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9lm\" (UniqueName: \"kubernetes.io/projected/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-kube-api-access-7f9lm\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.167557 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.167334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.268786 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.268743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.268786 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.268788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9lm\" (UniqueName: \"kubernetes.io/projected/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-kube-api-access-7f9lm\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.269062 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.268837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.269414 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.269392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.271464 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.271442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.286052 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.285990 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9lm\" (UniqueName: \"kubernetes.io/projected/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-kube-api-access-7f9lm\") pod \"maas-api-e2e-shared-b-de2394-b45df858f-dqqkr\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.341182 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.341140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:40.488565 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.488512 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr"] Aug 04 18:58:40.491645 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:58:40.491615 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d91e22_7d94_42c3_8b41_e74e0dd8b6aa.slice/crio-5e3a89fb4052ca1a9e8a21b7509711617a6f3df7f7fceceb291dae6edb078c16 WatchSource:0}: Error finding container 5e3a89fb4052ca1a9e8a21b7509711617a6f3df7f7fceceb291dae6edb078c16: Status 404 returned error can't find the container with id 5e3a89fb4052ca1a9e8a21b7509711617a6f3df7f7fceceb291dae6edb078c16 Aug 04 18:58:40.700401 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.700344 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s"] Aug 04 18:58:40.705965 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.705943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.708702 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.708679 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-de2394-dockercfg-jj29h\"" Aug 04 18:58:40.708840 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.708681 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-de2394\"" Aug 04 18:58:40.714444 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.714410 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s"] Aug 04 18:58:40.742859 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.742823 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b"] Aug 04 18:58:40.746297 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.746273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:40.757096 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.757054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b"] Aug 04 18:58:40.774837 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.774799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c588bf-3b23-4ec3-91f7-fc6436d22e63-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.775078 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.774905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7n9\" (UniqueName: \"kubernetes.io/projected/27c588bf-3b23-4ec3-91f7-fc6436d22e63-kube-api-access-zr7n9\") pod \"payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.875879 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.875834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2l9q\" (UniqueName: \"kubernetes.io/projected/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-kube-api-access-g2l9q\") pod \"payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:40.876112 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.875887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c588bf-3b23-4ec3-91f7-fc6436d22e63-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.876112 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.875926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:40.876112 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.875996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7n9\" (UniqueName: \"kubernetes.io/projected/27c588bf-3b23-4ec3-91f7-fc6436d22e63-kube-api-access-zr7n9\") pod \"payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.876854 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.876826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c588bf-3b23-4ec3-91f7-fc6436d22e63-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.886096 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.886065 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7n9\" (UniqueName: \"kubernetes.io/projected/27c588bf-3b23-4ec3-91f7-fc6436d22e63-kube-api-access-zr7n9\") pod \"payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:40.976738 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.976668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2l9q\" (UniqueName: \"kubernetes.io/projected/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-kube-api-access-g2l9q\") pod \"payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:40.976738 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.976712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:40.977534 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.977509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:40.985644 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:40.985617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2l9q\" (UniqueName: \"kubernetes.io/projected/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-kube-api-access-g2l9q\") pod \"payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:41.020207 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.020181 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:41.059301 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.059262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:41.177142 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.177057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s"] Aug 04 18:58:41.179543 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:58:41.179513 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c588bf_3b23_4ec3_91f7_fc6436d22e63.slice/crio-42f419833aaa0be49c01faeee0f7345381265be71bd35e82c1e89a343565d363 WatchSource:0}: Error finding container 42f419833aaa0be49c01faeee0f7345381265be71bd35e82c1e89a343565d363: Status 404 returned error can't find the container with id 42f419833aaa0be49c01faeee0f7345381265be71bd35e82c1e89a343565d363 Aug 04 18:58:41.229159 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.229133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b"] Aug 04 18:58:41.230608 ip-10-0-142-222 kubenswrapper[2575]: W0804 18:58:41.230579 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5e3c88_97fa_4c4a_a426_3b3d68e37df7.slice/crio-b7374042515a1f4421fe54f0561ea49cd12ac18254c9b34053aa698d06f78eab WatchSource:0}: Error finding container b7374042515a1f4421fe54f0561ea49cd12ac18254c9b34053aa698d06f78eab: Status 404 returned error can't find the container with id b7374042515a1f4421fe54f0561ea49cd12ac18254c9b34053aa698d06f78eab Aug 04 18:58:41.502524 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.502410 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" event={"ID":"27c588bf-3b23-4ec3-91f7-fc6436d22e63","Type":"ContainerStarted","Data":"4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5"} Aug 04 18:58:41.502690 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.502638 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:41.502765 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.502686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" event={"ID":"27c588bf-3b23-4ec3-91f7-fc6436d22e63","Type":"ContainerStarted","Data":"42f419833aaa0be49c01faeee0f7345381265be71bd35e82c1e89a343565d363"} Aug 04 18:58:41.504093 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.504064 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" event={"ID":"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7","Type":"ContainerStarted","Data":"ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250"} Aug 04 18:58:41.504231 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.504113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" event={"ID":"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7","Type":"ContainerStarted","Data":"b7374042515a1f4421fe54f0561ea49cd12ac18254c9b34053aa698d06f78eab"} Aug 04 18:58:41.504231 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.504157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 18:58:41.505679 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.505606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" event={"ID":"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa","Type":"ContainerStarted","Data":"c1ce60f05477afb9def6f2408fd1e207bba182a6db090f0d758a38bd043c2d5f"} Aug 04 18:58:41.505804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.505691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" event={"ID":"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa","Type":"ContainerStarted","Data":"5e3a89fb4052ca1a9e8a21b7509711617a6f3df7f7fceceb291dae6edb078c16"} Aug 04 18:58:41.505804 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.505740 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:41.521091 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.521039 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" podStartSLOduration=1.521020573 podStartE2EDuration="1.521020573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:41.519467084 +0000 UTC m=+2886.666283344" watchObservedRunningTime="2026-08-04 18:58:41.521020573 +0000 UTC m=+2886.667836866" Aug 04 18:58:41.542037 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.541977 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" podStartSLOduration=1.04807678 podStartE2EDuration="1.541960819s" totalImagesPullingTime="493.884039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:41.538723359 +0000 UTC m=+2886.685539619" watchObservedRunningTime="2026-08-04 18:58:41.541960819 +0000 UTC m=+2886.688777146" Aug 04 18:58:41.558212 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:41.558160 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" podStartSLOduration=1.5581401499999998 podStartE2EDuration="1.55814015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:58:41.556146012 +0000 UTC m=+2886.702962275" watchObservedRunningTime="2026-08-04 18:58:41.55814015 +0000 UTC m=+2886.704956408" Aug 04 18:58:47.517491 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:47.517454 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 18:58:52.514861 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:52.514830 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 18:58:52.515288 ip-10-0-142-222 kubenswrapper[2575]: I0804 18:58:52.515206 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 19:00:35.587253 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:00:35.587219 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:00:35.598168 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:00:35.598145 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:05:35.622377 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:05:35.622335 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:05:35.635495 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:05:35.635473 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:07:52.865835 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.865798 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d"] Aug 04 19:07:52.870200 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.870173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:52.872512 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.872485 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0c0dc7-serving-cert\"" Aug 04 19:07:52.882179 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.882152 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d"] Aug 04 19:07:52.952666 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.952633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:52.952666 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.952668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4272d631-b4f6-4c4d-968c-d68f95da1407-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:52.952866 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:52.952688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzrz\" (UniqueName: \"kubernetes.io/projected/4272d631-b4f6-4c4d-968c-d68f95da1407-kube-api-access-zpzrz\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.054000 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.053960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.054000 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.054003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4272d631-b4f6-4c4d-968c-d68f95da1407-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.054203 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.054024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzrz\" (UniqueName: \"kubernetes.io/projected/4272d631-b4f6-4c4d-968c-d68f95da1407-kube-api-access-zpzrz\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.054203 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:07:53.054129 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-serving-cert: secret "maas-api-e2e-ipp-a-0c0dc7-serving-cert" not found Aug 04 19:07:53.054203 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:07:53.054202 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls podName:4272d631-b4f6-4c4d-968c-d68f95da1407 nodeName:}" failed. No retries permitted until 2026-08-04 19:07:53.55418529 +0000 UTC m=+3438.701001528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls") pod "maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" (UID: "4272d631-b4f6-4c4d-968c-d68f95da1407") : secret "maas-api-e2e-ipp-a-0c0dc7-serving-cert" not found Aug 04 19:07:53.054635 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.054617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4272d631-b4f6-4c4d-968c-d68f95da1407-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.066016 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.065992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzrz\" (UniqueName: \"kubernetes.io/projected/4272d631-b4f6-4c4d-968c-d68f95da1407-kube-api-access-zpzrz\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.557981 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.557952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:53.558155 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:07:53.558101 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-serving-cert: secret "maas-api-e2e-ipp-a-0c0dc7-serving-cert" not found Aug 04 19:07:53.558195 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:07:53.558190 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls podName:4272d631-b4f6-4c4d-968c-d68f95da1407 nodeName:}" failed. No retries permitted until 2026-08-04 19:07:54.558172049 +0000 UTC m=+3439.704988286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls") pod "maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" (UID: "4272d631-b4f6-4c4d-968c-d68f95da1407") : secret "maas-api-e2e-ipp-a-0c0dc7-serving-cert" not found Aug 04 19:07:53.788251 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.788218 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6"] Aug 04 19:07:53.791753 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.791735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:53.794660 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.794642 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0c0dc7-dockercfg-dr8nd\"" Aug 04 19:07:53.797802 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.797781 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0c0dc7\"" Aug 04 19:07:53.802833 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.802810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6"] Aug 04 19:07:53.860898 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.860822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:53.860898 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.860873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzgn\" (UniqueName: \"kubernetes.io/projected/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-kube-api-access-vwzgn\") pod \"payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:53.938505 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.938474 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl"] Aug 04 19:07:53.941846 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.941831 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:53.961598 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.961572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjj5m\" (UniqueName: \"kubernetes.io/projected/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-kube-api-access-pjj5m\") pod \"payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:53.961710 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.961607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:53.961710 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.961641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzgn\" (UniqueName: \"kubernetes.io/projected/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-kube-api-access-vwzgn\") pod \"payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:53.961710 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.961668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:53.962231 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.962211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:53.978418 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:53.978392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzgn\" (UniqueName: \"kubernetes.io/projected/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-kube-api-access-vwzgn\") pod \"payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:54.050826 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.050802 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl"] Aug 04 19:07:54.062380 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.062337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj5m\" (UniqueName: \"kubernetes.io/projected/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-kube-api-access-pjj5m\") pod \"payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:54.062480 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.062408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:54.063006 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.062984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:54.081473 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.081444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjj5m\" (UniqueName: \"kubernetes.io/projected/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-kube-api-access-pjj5m\") pod \"payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:54.102784 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.102760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:54.251505 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.251476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:54.256896 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.256872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6"] Aug 04 19:07:54.264380 ip-10-0-142-222 kubenswrapper[2575]: W0804 19:07:54.264332 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a723d6e_8dc1_42ee_a7f1_22e9e7cda459.slice/crio-84ab5f290abd78255c837a0e227e5e9d26d83e77b928f4ffb6099f1b4e42d425 WatchSource:0}: Error finding container 84ab5f290abd78255c837a0e227e5e9d26d83e77b928f4ffb6099f1b4e42d425: Status 404 returned error can't find the container with id 84ab5f290abd78255c837a0e227e5e9d26d83e77b928f4ffb6099f1b4e42d425 Aug 04 19:07:54.399150 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.399121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl"] Aug 04 19:07:54.399429 ip-10-0-142-222 kubenswrapper[2575]: W0804 19:07:54.399407 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1723f4_0e6a_4d66_8ee6_8934fd518a02.slice/crio-1498fed4681d2d6111768e8c931053640b227489e997f6625e1656d4d1f4357a WatchSource:0}: Error finding container 1498fed4681d2d6111768e8c931053640b227489e997f6625e1656d4d1f4357a: Status 404 returned error can't find the container with id 1498fed4681d2d6111768e8c931053640b227489e997f6625e1656d4d1f4357a Aug 04 19:07:54.567158 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.567120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:54.569841 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.569809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:54.595329 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.595295 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" event={"ID":"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459","Type":"ContainerStarted","Data":"aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45"} Aug 04 19:07:54.595478 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.595375 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:07:54.595478 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.595401 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" event={"ID":"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459","Type":"ContainerStarted","Data":"84ab5f290abd78255c837a0e227e5e9d26d83e77b928f4ffb6099f1b4e42d425"} Aug 04 19:07:54.596819 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.596798 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" event={"ID":"1a1723f4-0e6a-4d66-8ee6-8934fd518a02","Type":"ContainerStarted","Data":"d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d"} Aug 04 19:07:54.596927 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.596844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" event={"ID":"1a1723f4-0e6a-4d66-8ee6-8934fd518a02","Type":"ContainerStarted","Data":"1498fed4681d2d6111768e8c931053640b227489e997f6625e1656d4d1f4357a"} Aug 04 19:07:54.596927 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.596870 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:07:54.614695 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.614632 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" podStartSLOduration=1.614618356 podStartE2EDuration="1.614618356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:54.61326861 +0000 UTC m=+3439.760084885" watchObservedRunningTime="2026-08-04 19:07:54.614618356 +0000 UTC m=+3439.761434614" Aug 04 19:07:54.634402 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.634342 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" podStartSLOduration=1.634328658 podStartE2EDuration="1.634328658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:54.632535638 +0000 UTC m=+3439.779351895" watchObservedRunningTime="2026-08-04 19:07:54.634328658 +0000 UTC m=+3439.781144951" Aug 04 19:07:54.681708 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.681683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:54.831301 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.829605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d"] Aug 04 19:07:54.836461 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:54.836431 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:07:55.605124 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:55.605035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" event={"ID":"4272d631-b4f6-4c4d-968c-d68f95da1407","Type":"ContainerStarted","Data":"52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282"} Aug 04 19:07:55.605124 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:55.605099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" event={"ID":"4272d631-b4f6-4c4d-968c-d68f95da1407","Type":"ContainerStarted","Data":"93483e74957b5f2eef2f7c9df1b14aa3c18ec650449b61da1f5b134675c48ee5"} Aug 04 19:07:55.605640 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:55.605221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:07:55.625849 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:07:55.625791 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" podStartSLOduration=3.161444455 podStartE2EDuration="3.625771511s" totalImagesPullingTime="464.327056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:55.623432468 +0000 UTC m=+3440.770248748" watchObservedRunningTime="2026-08-04 19:07:55.625771511 +0000 UTC m=+3440.772587770" Aug 04 19:08:01.616815 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:01.616779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:08:05.607820 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:05.607790 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:08:05.608188 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:05.607966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:08:17.683034 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.682999 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh"] Aug 04 19:08:17.687072 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.687050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:17.689895 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.689876 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4b6c58-dockercfg-nz5sl\"" Aug 04 19:08:17.689996 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.689894 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4b6c58\"" Aug 04 19:08:17.698621 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.698600 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh"] Aug 04 19:08:17.797161 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.797132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386b30fa-b42e-455d-97d1-a0665c1eb17a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:17.797315 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.797254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wtt\" (UniqueName: \"kubernetes.io/projected/386b30fa-b42e-455d-97d1-a0665c1eb17a-kube-api-access-n5wtt\") pod \"payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:17.898164 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.898125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386b30fa-b42e-455d-97d1-a0665c1eb17a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:17.898328 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.898209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wtt\" (UniqueName: \"kubernetes.io/projected/386b30fa-b42e-455d-97d1-a0665c1eb17a-kube-api-access-n5wtt\") pod \"payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:17.898861 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.898841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386b30fa-b42e-455d-97d1-a0665c1eb17a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:17.908651 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:17.908622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wtt\" (UniqueName: \"kubernetes.io/projected/386b30fa-b42e-455d-97d1-a0665c1eb17a-kube-api-access-n5wtt\") pod \"payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:18.000420 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:18.000337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:18.166053 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:18.166022 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh"] Aug 04 19:08:18.168516 ip-10-0-142-222 kubenswrapper[2575]: W0804 19:08:18.168483 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386b30fa_b42e_455d_97d1_a0665c1eb17a.slice/crio-24ae4d3c27cf436a90aafb0a983f636b482b02492d3497f96cecd3944c54d160 WatchSource:0}: Error finding container 24ae4d3c27cf436a90aafb0a983f636b482b02492d3497f96cecd3944c54d160: Status 404 returned error can't find the container with id 24ae4d3c27cf436a90aafb0a983f636b482b02492d3497f96cecd3944c54d160 Aug 04 19:08:18.738301 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:18.738259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" event={"ID":"386b30fa-b42e-455d-97d1-a0665c1eb17a","Type":"ContainerStarted","Data":"1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168"} Aug 04 19:08:18.738301 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:18.738308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" event={"ID":"386b30fa-b42e-455d-97d1-a0665c1eb17a","Type":"ContainerStarted","Data":"24ae4d3c27cf436a90aafb0a983f636b482b02492d3497f96cecd3944c54d160"} Aug 04 19:08:18.738817 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:18.738343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:18.759022 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:18.758963 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" podStartSLOduration=1.7589441460000002 podStartE2EDuration="1.758944146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:08:18.757640669 +0000 UTC m=+3463.904456967" watchObservedRunningTime="2026-08-04 19:08:18.758944146 +0000 UTC m=+3463.905760406" Aug 04 19:08:29.746690 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:29.746656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:08:42.234280 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:08:42.234245 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh_386b30fa-b42e-455d-97d1-a0665c1eb17a/payload-processing/0.log" Aug 04 19:09:20.052405 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:20.052358 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl_1a1723f4-0e6a-4d66-8ee6-8934fd518a02/payload-processing/0.log" Aug 04 19:09:20.327074 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:20.326994 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh_386b30fa-b42e-455d-97d1-a0665c1eb17a/payload-processing/0.log" Aug 04 19:09:28.776744 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.776705 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf"] Aug 04 19:09:28.780445 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.780427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:28.783353 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.783332 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c75b82\"" Aug 04 19:09:28.783628 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.783612 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c75b82-dockercfg-z2vxp\"" Aug 04 19:09:28.799737 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.799712 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf"] Aug 04 19:09:28.844180 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.844142 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb"] Aug 04 19:09:28.849947 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.849916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:28.850095 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.849953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjz5g\" (UniqueName: \"kubernetes.io/projected/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-kube-api-access-zjz5g\") pod \"payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:28.851412 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.851394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:28.856918 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.856885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb"] Aug 04 19:09:28.950629 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.950592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225jk\" (UniqueName: \"kubernetes.io/projected/c23bff92-8351-4d4c-9cec-0a9045d5106f-kube-api-access-225jk\") pod \"payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:28.950808 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.950650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:28.950808 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.950678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23bff92-8351-4d4c-9cec-0a9045d5106f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:28.950808 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.950764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjz5g\" (UniqueName: \"kubernetes.io/projected/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-kube-api-access-zjz5g\") pod \"payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:28.951349 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.951329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:28.965249 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:28.965225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjz5g\" (UniqueName: \"kubernetes.io/projected/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-kube-api-access-zjz5g\") pod \"payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:29.051728 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.051638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225jk\" (UniqueName: \"kubernetes.io/projected/c23bff92-8351-4d4c-9cec-0a9045d5106f-kube-api-access-225jk\") pod \"payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:29.051728 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.051687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23bff92-8351-4d4c-9cec-0a9045d5106f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:29.052340 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.052321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23bff92-8351-4d4c-9cec-0a9045d5106f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:29.065676 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.065649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-225jk\" (UniqueName: \"kubernetes.io/projected/c23bff92-8351-4d4c-9cec-0a9045d5106f-kube-api-access-225jk\") pod \"payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:29.090487 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.090456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:29.166856 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.166685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:29.325739 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.325708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb"] Aug 04 19:09:29.326897 ip-10-0-142-222 kubenswrapper[2575]: W0804 19:09:29.326875 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23bff92_8351_4d4c_9cec_0a9045d5106f.slice/crio-d2409af3fff175f545f2f56eb456943f45f796367780c7b7a87d11516ed25667 WatchSource:0}: Error finding container d2409af3fff175f545f2f56eb456943f45f796367780c7b7a87d11516ed25667: Status 404 returned error can't find the container with id d2409af3fff175f545f2f56eb456943f45f796367780c7b7a87d11516ed25667 Aug 04 19:09:29.456311 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:29.456284 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf"] Aug 04 19:09:29.457649 ip-10-0-142-222 kubenswrapper[2575]: W0804 19:09:29.457624 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dcc1a0_f8fd_48ba_96ea_3da3b2a2b95b.slice/crio-fcdaf13a4c5462bc11de46f6cc857bc4357322eeb2af439700d2f36555439a0a WatchSource:0}: Error finding container fcdaf13a4c5462bc11de46f6cc857bc4357322eeb2af439700d2f36555439a0a: Status 404 returned error can't find the container with id fcdaf13a4c5462bc11de46f6cc857bc4357322eeb2af439700d2f36555439a0a Aug 04 19:09:30.151392 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.151323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" event={"ID":"c23bff92-8351-4d4c-9cec-0a9045d5106f","Type":"ContainerStarted","Data":"f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584"} Aug 04 19:09:30.151809 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.151452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" event={"ID":"c23bff92-8351-4d4c-9cec-0a9045d5106f","Type":"ContainerStarted","Data":"d2409af3fff175f545f2f56eb456943f45f796367780c7b7a87d11516ed25667"} Aug 04 19:09:30.151809 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.151485 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:30.152975 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.152923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" event={"ID":"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b","Type":"ContainerStarted","Data":"e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886"} Aug 04 19:09:30.153086 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.152999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" event={"ID":"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b","Type":"ContainerStarted","Data":"fcdaf13a4c5462bc11de46f6cc857bc4357322eeb2af439700d2f36555439a0a"} Aug 04 19:09:30.153086 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.153021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:30.193611 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.193560 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" podStartSLOduration=2.193545765 podStartE2EDuration="2.193545765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:09:30.192479107 +0000 UTC m=+3535.339295365" watchObservedRunningTime="2026-08-04 19:09:30.193545765 +0000 UTC m=+3535.340362023" Aug 04 19:09:30.244416 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:30.244347 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" podStartSLOduration=2.244331941 podStartE2EDuration="2.244331941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:09:30.240926985 +0000 UTC m=+3535.387743242" watchObservedRunningTime="2026-08-04 19:09:30.244331941 +0000 UTC m=+3535.391148199" Aug 04 19:09:41.161571 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:41.161476 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:41.161571 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:41.161553 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:56.253163 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.253126 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb"] Aug 04 19:09:56.253718 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.253421 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" podUID="c23bff92-8351-4d4c-9cec-0a9045d5106f" containerName="payload-processing" containerID="cri-o://f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584" gracePeriod=30 Aug 04 19:09:56.257844 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.257810 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf"] Aug 04 19:09:56.258059 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.258010 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" podUID="90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" containerName="payload-pre-processing" containerID="cri-o://e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886" gracePeriod=30 Aug 04 19:09:56.543741 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.543716 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:56.560787 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.560765 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:56.717923 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.717892 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-plugins-config-volume\") pod \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " Aug 04 19:09:56.717923 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.717922 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23bff92-8351-4d4c-9cec-0a9045d5106f-plugins-config-volume\") pod \"c23bff92-8351-4d4c-9cec-0a9045d5106f\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " Aug 04 19:09:56.718174 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.717957 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-225jk\" (UniqueName: \"kubernetes.io/projected/c23bff92-8351-4d4c-9cec-0a9045d5106f-kube-api-access-225jk\") pod \"c23bff92-8351-4d4c-9cec-0a9045d5106f\" (UID: \"c23bff92-8351-4d4c-9cec-0a9045d5106f\") " Aug 04 19:09:56.718174 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.718012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjz5g\" (UniqueName: \"kubernetes.io/projected/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-kube-api-access-zjz5g\") pod \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\" (UID: \"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b\") " Aug 04 19:09:56.718324 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.718301 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" (UID: "90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:09:56.718450 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.718309 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23bff92-8351-4d4c-9cec-0a9045d5106f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c23bff92-8351-4d4c-9cec-0a9045d5106f" (UID: "c23bff92-8351-4d4c-9cec-0a9045d5106f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:09:56.718450 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.718419 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:09:56.718450 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.718439 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23bff92-8351-4d4c-9cec-0a9045d5106f-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:09:56.720350 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.720322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-kube-api-access-zjz5g" (OuterVolumeSpecName: "kube-api-access-zjz5g") pod "90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" (UID: "90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b"). InnerVolumeSpecName "kube-api-access-zjz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:09:56.720350 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.720347 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23bff92-8351-4d4c-9cec-0a9045d5106f-kube-api-access-225jk" (OuterVolumeSpecName: "kube-api-access-225jk") pod "c23bff92-8351-4d4c-9cec-0a9045d5106f" (UID: "c23bff92-8351-4d4c-9cec-0a9045d5106f"). InnerVolumeSpecName "kube-api-access-225jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:09:56.819418 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.819330 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-225jk\" (UniqueName: \"kubernetes.io/projected/c23bff92-8351-4d4c-9cec-0a9045d5106f-kube-api-access-225jk\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:09:56.819418 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:56.819353 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjz5g\" (UniqueName: \"kubernetes.io/projected/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b-kube-api-access-zjz5g\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:09:57.316447 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.316412 2575 generic.go:356] "Generic (PLEG): container finished" podID="c23bff92-8351-4d4c-9cec-0a9045d5106f" containerID="f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584" exitCode=0 Aug 04 19:09:57.316815 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.316453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" event={"ID":"c23bff92-8351-4d4c-9cec-0a9045d5106f","Type":"ContainerDied","Data":"f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584"} Aug 04 19:09:57.316815 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.316462 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" Aug 04 19:09:57.316815 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.316504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb" event={"ID":"c23bff92-8351-4d4c-9cec-0a9045d5106f","Type":"ContainerDied","Data":"d2409af3fff175f545f2f56eb456943f45f796367780c7b7a87d11516ed25667"} Aug 04 19:09:57.316815 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.316527 2575 scope.go:117] "RemoveContainer" containerID="f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584" Aug 04 19:09:57.318032 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.317990 2575 generic.go:356] "Generic (PLEG): container finished" podID="90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" containerID="e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886" exitCode=0 Aug 04 19:09:57.318032 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.318017 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" Aug 04 19:09:57.318160 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.318064 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" event={"ID":"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b","Type":"ContainerDied","Data":"e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886"} Aug 04 19:09:57.318160 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.318104 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf" event={"ID":"90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b","Type":"ContainerDied","Data":"fcdaf13a4c5462bc11de46f6cc857bc4357322eeb2af439700d2f36555439a0a"} Aug 04 19:09:57.326776 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.326760 2575 scope.go:117] "RemoveContainer" containerID="f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584" Aug 04 19:09:57.327065 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:09:57.327042 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584\": container with ID starting with f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584 not found: ID does not exist" containerID="f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584" Aug 04 19:09:57.327163 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.327075 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584"} err="failed to get container status \"f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584\": rpc error: code = NotFound desc = could not find container \"f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584\": container with ID starting with f15e5c2521ff84ae0d1035d2172df46e857baed1d37debe08ae00b1470211584 not found: ID does not exist" Aug 04 19:09:57.327163 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.327099 2575 scope.go:117] "RemoveContainer" containerID="e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886" Aug 04 19:09:57.336055 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.336038 2575 scope.go:117] "RemoveContainer" containerID="e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886" Aug 04 19:09:57.336286 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:09:57.336271 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886\": container with ID starting with e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886 not found: ID does not exist" containerID="e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886" Aug 04 19:09:57.336331 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.336293 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886"} err="failed to get container status \"e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886\": rpc error: code = NotFound desc = could not find container \"e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886\": container with ID starting with e69447ef0168aec13641dd12aabb2db4a992652342a94d74406dd5d6e442b886 not found: ID does not exist" Aug 04 19:09:57.349235 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.349214 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf"] Aug 04 19:09:57.354276 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.354253 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c75b82-5b6df957cb-7vblf"] Aug 04 19:09:57.367186 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.367162 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb"] Aug 04 19:09:57.374475 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:57.374452 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c75b82-dd48bb57d-lh9jb"] Aug 04 19:09:59.348565 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:59.348532 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" path="/var/lib/kubelet/pods/90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b/volumes" Aug 04 19:09:59.348948 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:09:59.348931 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c23bff92-8351-4d4c-9cec-0a9045d5106f" path="/var/lib/kubelet/pods/c23bff92-8351-4d4c-9cec-0a9045d5106f/volumes" Aug 04 19:10:28.568207 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.568128 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d"] Aug 04 19:10:28.568701 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.568343 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" podUID="4272d631-b4f6-4c4d-968c-d68f95da1407" containerName="maas-api" containerID="cri-o://52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282" gracePeriod=30 Aug 04 19:10:28.664844 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.664807 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl"] Aug 04 19:10:28.665137 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.665090 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" podUID="1a1723f4-0e6a-4d66-8ee6-8934fd518a02" containerName="payload-processing" containerID="cri-o://d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d" gracePeriod=30 Aug 04 19:10:28.695063 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.695031 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6"] Aug 04 19:10:28.695416 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.695273 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" podUID="0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" containerName="payload-pre-processing" containerID="cri-o://aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45" gracePeriod=30 Aug 04 19:10:28.852931 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.852903 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:10:28.971767 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.971740 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:10:28.976800 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:28.976780 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:10:29.017080 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.016965 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") pod \"4272d631-b4f6-4c4d-968c-d68f95da1407\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " Aug 04 19:10:29.017080 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.017026 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpzrz\" (UniqueName: \"kubernetes.io/projected/4272d631-b4f6-4c4d-968c-d68f95da1407-kube-api-access-zpzrz\") pod \"4272d631-b4f6-4c4d-968c-d68f95da1407\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " Aug 04 19:10:29.017080 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.017055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4272d631-b4f6-4c4d-968c-d68f95da1407-openshift-service-ca\") pod \"4272d631-b4f6-4c4d-968c-d68f95da1407\" (UID: \"4272d631-b4f6-4c4d-968c-d68f95da1407\") " Aug 04 19:10:29.017970 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.017938 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4272d631-b4f6-4c4d-968c-d68f95da1407-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4272d631-b4f6-4c4d-968c-d68f95da1407" (UID: "4272d631-b4f6-4c4d-968c-d68f95da1407"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:10:29.020047 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.020021 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4272d631-b4f6-4c4d-968c-d68f95da1407" (UID: "4272d631-b4f6-4c4d-968c-d68f95da1407"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:10:29.020134 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.020114 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4272d631-b4f6-4c4d-968c-d68f95da1407-kube-api-access-zpzrz" (OuterVolumeSpecName: "kube-api-access-zpzrz") pod "4272d631-b4f6-4c4d-968c-d68f95da1407" (UID: "4272d631-b4f6-4c4d-968c-d68f95da1407"). InnerVolumeSpecName "kube-api-access-zpzrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:10:29.118002 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.117916 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwzgn\" (UniqueName: \"kubernetes.io/projected/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-kube-api-access-vwzgn\") pod \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " Aug 04 19:10:29.118002 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.117952 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-plugins-config-volume\") pod \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " Aug 04 19:10:29.118002 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.117973 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjj5m\" (UniqueName: \"kubernetes.io/projected/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-kube-api-access-pjj5m\") pod \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\" (UID: \"1a1723f4-0e6a-4d66-8ee6-8934fd518a02\") " Aug 04 19:10:29.118002 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.118001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-plugins-config-volume\") pod \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\" (UID: \"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459\") " Aug 04 19:10:29.118334 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.118285 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4272d631-b4f6-4c4d-968c-d68f95da1407-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.118334 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.118306 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpzrz\" (UniqueName: \"kubernetes.io/projected/4272d631-b4f6-4c4d-968c-d68f95da1407-kube-api-access-zpzrz\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.118334 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.118320 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4272d631-b4f6-4c4d-968c-d68f95da1407-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.118547 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.118356 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a1723f4-0e6a-4d66-8ee6-8934fd518a02" (UID: "1a1723f4-0e6a-4d66-8ee6-8934fd518a02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:10:29.118547 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.118393 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" (UID: "0a723d6e-8dc1-42ee-a7f1-22e9e7cda459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:10:29.120135 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.120113 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-kube-api-access-pjj5m" (OuterVolumeSpecName: "kube-api-access-pjj5m") pod "1a1723f4-0e6a-4d66-8ee6-8934fd518a02" (UID: "1a1723f4-0e6a-4d66-8ee6-8934fd518a02"). InnerVolumeSpecName "kube-api-access-pjj5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:10:29.120202 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.120134 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-kube-api-access-vwzgn" (OuterVolumeSpecName: "kube-api-access-vwzgn") pod "0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" (UID: "0a723d6e-8dc1-42ee-a7f1-22e9e7cda459"). InnerVolumeSpecName "kube-api-access-vwzgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:10:29.219737 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.219698 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwzgn\" (UniqueName: \"kubernetes.io/projected/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-kube-api-access-vwzgn\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.219737 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.219729 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.219737 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.219739 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjj5m\" (UniqueName: \"kubernetes.io/projected/1a1723f4-0e6a-4d66-8ee6-8934fd518a02-kube-api-access-pjj5m\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.219737 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.219748 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:10:29.515676 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.515639 2575 generic.go:356] "Generic (PLEG): container finished" podID="0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" containerID="aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45" exitCode=0 Aug 04 19:10:29.515832 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.515674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" event={"ID":"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459","Type":"ContainerDied","Data":"aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45"} Aug 04 19:10:29.515832 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.515729 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" event={"ID":"0a723d6e-8dc1-42ee-a7f1-22e9e7cda459","Type":"ContainerDied","Data":"84ab5f290abd78255c837a0e227e5e9d26d83e77b928f4ffb6099f1b4e42d425"} Aug 04 19:10:29.515832 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.515686 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6" Aug 04 19:10:29.515832 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.515748 2575 scope.go:117] "RemoveContainer" containerID="aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45" Aug 04 19:10:29.517259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.517118 2575 generic.go:356] "Generic (PLEG): container finished" podID="4272d631-b4f6-4c4d-968c-d68f95da1407" containerID="52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282" exitCode=0 Aug 04 19:10:29.517259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.517229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" event={"ID":"4272d631-b4f6-4c4d-968c-d68f95da1407","Type":"ContainerDied","Data":"52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282"} Aug 04 19:10:29.517431 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.517281 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" event={"ID":"4272d631-b4f6-4c4d-968c-d68f95da1407","Type":"ContainerDied","Data":"93483e74957b5f2eef2f7c9df1b14aa3c18ec650449b61da1f5b134675c48ee5"} Aug 04 19:10:29.517431 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.517317 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d" Aug 04 19:10:29.519015 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.518989 2575 generic.go:356] "Generic (PLEG): container finished" podID="1a1723f4-0e6a-4d66-8ee6-8934fd518a02" containerID="d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d" exitCode=0 Aug 04 19:10:29.519139 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.519032 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" Aug 04 19:10:29.519139 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.519064 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" event={"ID":"1a1723f4-0e6a-4d66-8ee6-8934fd518a02","Type":"ContainerDied","Data":"d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d"} Aug 04 19:10:29.519139 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.519134 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl" event={"ID":"1a1723f4-0e6a-4d66-8ee6-8934fd518a02","Type":"ContainerDied","Data":"1498fed4681d2d6111768e8c931053640b227489e997f6625e1656d4d1f4357a"} Aug 04 19:10:29.526190 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.526172 2575 scope.go:117] "RemoveContainer" containerID="aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45" Aug 04 19:10:29.526481 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:10:29.526455 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45\": container with ID starting with aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45 not found: ID does not exist" containerID="aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45" Aug 04 19:10:29.526560 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.526488 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45"} err="failed to get container status \"aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45\": rpc error: code = NotFound desc = could not find container \"aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45\": container with ID starting with aa2e6cf02121d57ac1abcb0e3d03c9d4b79cf78fb17730d7c936e1f721449d45 not found: ID does not exist" Aug 04 19:10:29.526560 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.526503 2575 scope.go:117] "RemoveContainer" containerID="52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282" Aug 04 19:10:29.534571 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.534551 2575 scope.go:117] "RemoveContainer" containerID="52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282" Aug 04 19:10:29.534810 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:10:29.534794 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282\": container with ID starting with 52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282 not found: ID does not exist" containerID="52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282" Aug 04 19:10:29.534855 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.534826 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282"} err="failed to get container status \"52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282\": rpc error: code = NotFound desc = could not find container \"52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282\": container with ID starting with 52f47543aa58964edcfb8d54bb75a5cc8e038f65717d6259592dd6f423af5282 not found: ID does not exist" Aug 04 19:10:29.534855 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.534841 2575 scope.go:117] "RemoveContainer" containerID="d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d" Aug 04 19:10:29.541685 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.541655 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6"] Aug 04 19:10:29.546390 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.546330 2575 scope.go:117] "RemoveContainer" containerID="d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d" Aug 04 19:10:29.546760 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:10:29.546734 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d\": container with ID starting with d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d not found: ID does not exist" containerID="d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d" Aug 04 19:10:29.546855 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.546769 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d"} err="failed to get container status \"d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d\": rpc error: code = NotFound desc = could not find container \"d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d\": container with ID starting with d38a928419cba744b391301e47c67a3637105b87a5bc359898b2a9acc0b71d2d not found: ID does not exist" Aug 04 19:10:29.549303 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.549277 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c0dc7-69778546dd-zl7n6"] Aug 04 19:10:29.558356 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.558334 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl"] Aug 04 19:10:29.562988 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.562966 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c0dc7-77bf969548-6z7zl"] Aug 04 19:10:29.586154 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.586131 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d"] Aug 04 19:10:29.594481 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:29.594463 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0c0dc7-6b774644c6-q7t5d"] Aug 04 19:10:31.349090 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:31.349054 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" path="/var/lib/kubelet/pods/0a723d6e-8dc1-42ee-a7f1-22e9e7cda459/volumes" Aug 04 19:10:31.349494 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:31.349475 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1723f4-0e6a-4d66-8ee6-8934fd518a02" path="/var/lib/kubelet/pods/1a1723f4-0e6a-4d66-8ee6-8934fd518a02/volumes" Aug 04 19:10:31.349819 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:31.349805 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4272d631-b4f6-4c4d-968c-d68f95da1407" path="/var/lib/kubelet/pods/4272d631-b4f6-4c4d-968c-d68f95da1407/volumes" Aug 04 19:10:35.666299 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:35.666263 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:10:35.682719 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:10:35.682694 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:11:34.204506 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.204467 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh"] Aug 04 19:11:34.205213 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.204743 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" podUID="386b30fa-b42e-455d-97d1-a0665c1eb17a" containerName="payload-processing" containerID="cri-o://1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168" gracePeriod=30 Aug 04 19:11:34.475317 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.475291 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:11:34.524996 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.524957 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5wtt\" (UniqueName: \"kubernetes.io/projected/386b30fa-b42e-455d-97d1-a0665c1eb17a-kube-api-access-n5wtt\") pod \"386b30fa-b42e-455d-97d1-a0665c1eb17a\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " Aug 04 19:11:34.525161 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.525018 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386b30fa-b42e-455d-97d1-a0665c1eb17a-plugins-config-volume\") pod \"386b30fa-b42e-455d-97d1-a0665c1eb17a\" (UID: \"386b30fa-b42e-455d-97d1-a0665c1eb17a\") " Aug 04 19:11:34.525421 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.525395 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386b30fa-b42e-455d-97d1-a0665c1eb17a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "386b30fa-b42e-455d-97d1-a0665c1eb17a" (UID: "386b30fa-b42e-455d-97d1-a0665c1eb17a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:34.527403 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.527343 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386b30fa-b42e-455d-97d1-a0665c1eb17a-kube-api-access-n5wtt" (OuterVolumeSpecName: "kube-api-access-n5wtt") pod "386b30fa-b42e-455d-97d1-a0665c1eb17a" (UID: "386b30fa-b42e-455d-97d1-a0665c1eb17a"). InnerVolumeSpecName "kube-api-access-n5wtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:34.626181 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.626142 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5wtt\" (UniqueName: \"kubernetes.io/projected/386b30fa-b42e-455d-97d1-a0665c1eb17a-kube-api-access-n5wtt\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.626181 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.626178 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386b30fa-b42e-455d-97d1-a0665c1eb17a-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:34.889917 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.889827 2575 generic.go:356] "Generic (PLEG): container finished" podID="386b30fa-b42e-455d-97d1-a0665c1eb17a" containerID="1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168" exitCode=0 Aug 04 19:11:34.889917 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.889873 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" Aug 04 19:11:34.890111 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.889915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" event={"ID":"386b30fa-b42e-455d-97d1-a0665c1eb17a","Type":"ContainerDied","Data":"1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168"} Aug 04 19:11:34.890111 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.889990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh" event={"ID":"386b30fa-b42e-455d-97d1-a0665c1eb17a","Type":"ContainerDied","Data":"24ae4d3c27cf436a90aafb0a983f636b482b02492d3497f96cecd3944c54d160"} Aug 04 19:11:34.890111 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.890012 2575 scope.go:117] "RemoveContainer" containerID="1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168" Aug 04 19:11:34.899518 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.899501 2575 scope.go:117] "RemoveContainer" containerID="1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168" Aug 04 19:11:34.899759 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:11:34.899743 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168\": container with ID starting with 1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168 not found: ID does not exist" containerID="1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168" Aug 04 19:11:34.899826 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.899765 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168"} err="failed to get container status \"1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168\": rpc error: code = NotFound desc = could not find container \"1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168\": container with ID starting with 1f531e86923d0ebc2f039d73abe5e825b24ea05d1233198b1d05c460403ff168 not found: ID does not exist" Aug 04 19:11:34.923647 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.923608 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh"] Aug 04 19:11:34.928975 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:34.928951 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4b6c58-5cb645fc76-6qkhh"] Aug 04 19:11:35.350611 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:35.350575 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386b30fa-b42e-455d-97d1-a0665c1eb17a" path="/var/lib/kubelet/pods/386b30fa-b42e-455d-97d1-a0665c1eb17a/volumes" Aug 04 19:11:50.384909 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.384874 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8"] Aug 04 19:11:50.385342 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.385134 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" podUID="06ff10c0-2667-47c7-abef-09728d4ece78" containerName="maas-api" containerID="cri-o://d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf" gracePeriod=30 Aug 04 19:11:50.479769 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.479734 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54"] Aug 04 19:11:50.479986 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.479950 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" podUID="105a19a6-4014-48ec-8343-b705bc7b5dbe" containerName="payload-processing" containerID="cri-o://ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943" gracePeriod=30 Aug 04 19:11:50.489431 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.489404 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g"] Aug 04 19:11:50.489832 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.489801 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" podUID="34710ed6-65be-4d7c-8b50-2c4d3d581f83" containerName="payload-pre-processing" containerID="cri-o://a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a" gracePeriod=30 Aug 04 19:11:50.711302 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.711271 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 19:11:50.742133 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.742106 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 19:11:50.781857 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.781816 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ff10c0-2667-47c7-abef-09728d4ece78-openshift-service-ca\") pod \"06ff10c0-2667-47c7-abef-09728d4ece78\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " Aug 04 19:11:50.782071 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.781904 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ff10c0-2667-47c7-abef-09728d4ece78-maas-api-tls\") pod \"06ff10c0-2667-47c7-abef-09728d4ece78\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " Aug 04 19:11:50.782071 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.781979 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzl94\" (UniqueName: \"kubernetes.io/projected/06ff10c0-2667-47c7-abef-09728d4ece78-kube-api-access-zzl94\") pod \"06ff10c0-2667-47c7-abef-09728d4ece78\" (UID: \"06ff10c0-2667-47c7-abef-09728d4ece78\") " Aug 04 19:11:50.782199 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.782141 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ff10c0-2667-47c7-abef-09728d4ece78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06ff10c0-2667-47c7-abef-09728d4ece78" (UID: "06ff10c0-2667-47c7-abef-09728d4ece78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:50.782321 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.782301 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ff10c0-2667-47c7-abef-09728d4ece78-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.784413 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.784380 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ff10c0-2667-47c7-abef-09728d4ece78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06ff10c0-2667-47c7-abef-09728d4ece78" (UID: "06ff10c0-2667-47c7-abef-09728d4ece78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:11:50.784606 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.784587 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ff10c0-2667-47c7-abef-09728d4ece78-kube-api-access-zzl94" (OuterVolumeSpecName: "kube-api-access-zzl94") pod "06ff10c0-2667-47c7-abef-09728d4ece78" (UID: "06ff10c0-2667-47c7-abef-09728d4ece78"). InnerVolumeSpecName "kube-api-access-zzl94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:50.785454 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.785437 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 19:11:50.883150 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883118 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105a19a6-4014-48ec-8343-b705bc7b5dbe-plugins-config-volume\") pod \"105a19a6-4014-48ec-8343-b705bc7b5dbe\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " Aug 04 19:11:50.883323 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883192 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25wkr\" (UniqueName: \"kubernetes.io/projected/34710ed6-65be-4d7c-8b50-2c4d3d581f83-kube-api-access-25wkr\") pod \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " Aug 04 19:11:50.883323 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34710ed6-65be-4d7c-8b50-2c4d3d581f83-plugins-config-volume\") pod \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\" (UID: \"34710ed6-65be-4d7c-8b50-2c4d3d581f83\") " Aug 04 19:11:50.883323 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883259 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qznzk\" (UniqueName: \"kubernetes.io/projected/105a19a6-4014-48ec-8343-b705bc7b5dbe-kube-api-access-qznzk\") pod \"105a19a6-4014-48ec-8343-b705bc7b5dbe\" (UID: \"105a19a6-4014-48ec-8343-b705bc7b5dbe\") " Aug 04 19:11:50.883511 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883494 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ff10c0-2667-47c7-abef-09728d4ece78-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.883590 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883520 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzl94\" (UniqueName: \"kubernetes.io/projected/06ff10c0-2667-47c7-abef-09728d4ece78-kube-api-access-zzl94\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.883590 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883545 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105a19a6-4014-48ec-8343-b705bc7b5dbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "105a19a6-4014-48ec-8343-b705bc7b5dbe" (UID: "105a19a6-4014-48ec-8343-b705bc7b5dbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:50.883683 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.883638 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34710ed6-65be-4d7c-8b50-2c4d3d581f83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34710ed6-65be-4d7c-8b50-2c4d3d581f83" (UID: "34710ed6-65be-4d7c-8b50-2c4d3d581f83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:11:50.885503 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.885479 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34710ed6-65be-4d7c-8b50-2c4d3d581f83-kube-api-access-25wkr" (OuterVolumeSpecName: "kube-api-access-25wkr") pod "34710ed6-65be-4d7c-8b50-2c4d3d581f83" (UID: "34710ed6-65be-4d7c-8b50-2c4d3d581f83"). InnerVolumeSpecName "kube-api-access-25wkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:50.885627 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.885609 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105a19a6-4014-48ec-8343-b705bc7b5dbe-kube-api-access-qznzk" (OuterVolumeSpecName: "kube-api-access-qznzk") pod "105a19a6-4014-48ec-8343-b705bc7b5dbe" (UID: "105a19a6-4014-48ec-8343-b705bc7b5dbe"). InnerVolumeSpecName "kube-api-access-qznzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:11:50.986627 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.986590 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/105a19a6-4014-48ec-8343-b705bc7b5dbe-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.986824 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.986627 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wkr\" (UniqueName: \"kubernetes.io/projected/34710ed6-65be-4d7c-8b50-2c4d3d581f83-kube-api-access-25wkr\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.986871 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.986819 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34710ed6-65be-4d7c-8b50-2c4d3d581f83-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.986871 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.986837 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznzk\" (UniqueName: \"kubernetes.io/projected/105a19a6-4014-48ec-8343-b705bc7b5dbe-kube-api-access-qznzk\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:11:50.987687 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.987659 2575 generic.go:356] "Generic (PLEG): container finished" podID="06ff10c0-2667-47c7-abef-09728d4ece78" containerID="d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf" exitCode=0 Aug 04 19:11:50.987811 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.987715 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" Aug 04 19:11:50.987811 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.987750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" event={"ID":"06ff10c0-2667-47c7-abef-09728d4ece78","Type":"ContainerDied","Data":"d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf"} Aug 04 19:11:50.987904 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.987835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8" event={"ID":"06ff10c0-2667-47c7-abef-09728d4ece78","Type":"ContainerDied","Data":"dc9386276a4727580d456436e269b4f528cfe369e4b54717cbb8196261b87f25"} Aug 04 19:11:50.987904 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.987862 2575 scope.go:117] "RemoveContainer" containerID="d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf" Aug 04 19:11:50.989173 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.989153 2575 generic.go:356] "Generic (PLEG): container finished" podID="34710ed6-65be-4d7c-8b50-2c4d3d581f83" containerID="a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a" exitCode=0 Aug 04 19:11:50.989262 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.989241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" Aug 04 19:11:50.989315 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.989269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" event={"ID":"34710ed6-65be-4d7c-8b50-2c4d3d581f83","Type":"ContainerDied","Data":"a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a"} Aug 04 19:11:50.989315 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.989311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g" event={"ID":"34710ed6-65be-4d7c-8b50-2c4d3d581f83","Type":"ContainerDied","Data":"6eee7965e8ba0aa1873c5260c6838eb0799a8aeb8505c7eb054f43d1ddbd789e"} Aug 04 19:11:50.990587 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.990565 2575 generic.go:356] "Generic (PLEG): container finished" podID="105a19a6-4014-48ec-8343-b705bc7b5dbe" containerID="ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943" exitCode=0 Aug 04 19:11:50.990680 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.990599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" event={"ID":"105a19a6-4014-48ec-8343-b705bc7b5dbe","Type":"ContainerDied","Data":"ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943"} Aug 04 19:11:50.990680 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.990635 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" event={"ID":"105a19a6-4014-48ec-8343-b705bc7b5dbe","Type":"ContainerDied","Data":"6ba7683dcb5111e1478b92f1cd9b43b272ff5b4b6abe9c7fd324c7f1e8bfdf27"} Aug 04 19:11:50.990680 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:50.990642 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54" Aug 04 19:11:51.002141 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.002116 2575 scope.go:117] "RemoveContainer" containerID="d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf" Aug 04 19:11:51.002684 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:11:51.002419 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf\": container with ID starting with d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf not found: ID does not exist" containerID="d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf" Aug 04 19:11:51.002684 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.002456 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf"} err="failed to get container status \"d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf\": rpc error: code = NotFound desc = could not find container \"d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf\": container with ID starting with d909528fc55ccb079120daa90e884089bb6678c2d871b582ebe0f21961718ebf not found: ID does not exist" Aug 04 19:11:51.002684 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.002481 2575 scope.go:117] "RemoveContainer" containerID="a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a" Aug 04 19:11:51.011038 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.011022 2575 scope.go:117] "RemoveContainer" containerID="a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a" Aug 04 19:11:51.011294 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:11:51.011276 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a\": container with ID starting with a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a not found: ID does not exist" containerID="a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a" Aug 04 19:11:51.011390 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.011304 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a"} err="failed to get container status \"a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a\": rpc error: code = NotFound desc = could not find container \"a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a\": container with ID starting with a5caa2b26a330a4a4ee37d7bd84f3c82ccb79af5aabb463ba8338afba907290a not found: ID does not exist" Aug 04 19:11:51.011390 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.011326 2575 scope.go:117] "RemoveContainer" containerID="ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943" Aug 04 19:11:51.019522 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.019504 2575 scope.go:117] "RemoveContainer" containerID="ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943" Aug 04 19:11:51.019809 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:11:51.019783 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943\": container with ID starting with ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943 not found: ID does not exist" containerID="ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943" Aug 04 19:11:51.019905 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.019815 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943"} err="failed to get container status \"ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943\": rpc error: code = NotFound desc = could not find container \"ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943\": container with ID starting with ad0dc2c6fdeb815611e272d7358646d4ea28f0bdd950b016d42e1c46e86f7943 not found: ID does not exist" Aug 04 19:11:51.020571 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.020552 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g"] Aug 04 19:11:51.025627 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.025602 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-219d48-cc9548b5b-6dm5g"] Aug 04 19:11:51.037184 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.037152 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54"] Aug 04 19:11:51.039817 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.039795 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-219d48-6cbfcb5ddb-pnh54"] Aug 04 19:11:51.052179 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.052158 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8"] Aug 04 19:11:51.054223 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.054204 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-219d48-59b7f49b67-blsv8"] Aug 04 19:11:51.349327 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.349243 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ff10c0-2667-47c7-abef-09728d4ece78" path="/var/lib/kubelet/pods/06ff10c0-2667-47c7-abef-09728d4ece78/volumes" Aug 04 19:11:51.349655 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.349640 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105a19a6-4014-48ec-8343-b705bc7b5dbe" path="/var/lib/kubelet/pods/105a19a6-4014-48ec-8343-b705bc7b5dbe/volumes" Aug 04 19:11:51.349993 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:11:51.349972 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34710ed6-65be-4d7c-8b50-2c4d3d581f83" path="/var/lib/kubelet/pods/34710ed6-65be-4d7c-8b50-2c4d3d581f83/volumes" Aug 04 19:12:04.861639 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:04.861561 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr"] Aug 04 19:12:04.862065 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:04.861798 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" podUID="e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" containerName="maas-api" containerID="cri-o://c1ce60f05477afb9def6f2408fd1e207bba182a6db090f0d758a38bd043c2d5f" gracePeriod=30 Aug 04 19:12:05.031014 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.030960 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s"] Aug 04 19:12:05.031592 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.031556 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" podUID="27c588bf-3b23-4ec3-91f7-fc6436d22e63" containerName="payload-processing" containerID="cri-o://4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5" gracePeriod=30 Aug 04 19:12:05.040821 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.040793 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b"] Aug 04 19:12:05.041008 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.040982 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" podUID="9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" containerName="payload-pre-processing" containerID="cri-o://ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250" gracePeriod=30 Aug 04 19:12:05.075086 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.074968 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" containerID="c1ce60f05477afb9def6f2408fd1e207bba182a6db090f0d758a38bd043c2d5f" exitCode=0 Aug 04 19:12:05.075086 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.075014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" event={"ID":"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa","Type":"ContainerDied","Data":"c1ce60f05477afb9def6f2408fd1e207bba182a6db090f0d758a38bd043c2d5f"} Aug 04 19:12:05.163980 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.163943 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 19:12:05.312190 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.312167 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 19:12:05.314765 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.314743 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 19:12:05.318985 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.318963 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-openshift-service-ca\") pod \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " Aug 04 19:12:05.319061 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.318993 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f9lm\" (UniqueName: \"kubernetes.io/projected/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-kube-api-access-7f9lm\") pod \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " Aug 04 19:12:05.319116 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.319099 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-maas-api-tls\") pod \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\" (UID: \"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa\") " Aug 04 19:12:05.319300 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.319275 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" (UID: "e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:12:05.321061 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.321034 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-kube-api-access-7f9lm" (OuterVolumeSpecName: "kube-api-access-7f9lm") pod "e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" (UID: "e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa"). InnerVolumeSpecName "kube-api-access-7f9lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:05.321134 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.321076 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" (UID: "e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:12:05.420528 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr7n9\" (UniqueName: \"kubernetes.io/projected/27c588bf-3b23-4ec3-91f7-fc6436d22e63-kube-api-access-zr7n9\") pod \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " Aug 04 19:12:05.420659 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420544 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c588bf-3b23-4ec3-91f7-fc6436d22e63-plugins-config-volume\") pod \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\" (UID: \"27c588bf-3b23-4ec3-91f7-fc6436d22e63\") " Aug 04 19:12:05.420659 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420610 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2l9q\" (UniqueName: \"kubernetes.io/projected/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-kube-api-access-g2l9q\") pod \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " Aug 04 19:12:05.420659 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420651 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-plugins-config-volume\") pod \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\" (UID: \"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7\") " Aug 04 19:12:05.420977 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420959 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-maas-api-tls\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:05.421055 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420984 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-openshift-service-ca\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:05.421055 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420989 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c588bf-3b23-4ec3-91f7-fc6436d22e63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27c588bf-3b23-4ec3-91f7-fc6436d22e63" (UID: "27c588bf-3b23-4ec3-91f7-fc6436d22e63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:12:05.421055 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.420996 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9lm\" (UniqueName: \"kubernetes.io/projected/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa-kube-api-access-7f9lm\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:05.421151 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.421065 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" (UID: "9e5e3c88-97fa-4c4a-a426-3b3d68e37df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:12:05.422729 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.422711 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c588bf-3b23-4ec3-91f7-fc6436d22e63-kube-api-access-zr7n9" (OuterVolumeSpecName: "kube-api-access-zr7n9") pod "27c588bf-3b23-4ec3-91f7-fc6436d22e63" (UID: "27c588bf-3b23-4ec3-91f7-fc6436d22e63"). InnerVolumeSpecName "kube-api-access-zr7n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:05.422830 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.422811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-kube-api-access-g2l9q" (OuterVolumeSpecName: "kube-api-access-g2l9q") pod "9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" (UID: "9e5e3c88-97fa-4c4a-a426-3b3d68e37df7"). InnerVolumeSpecName "kube-api-access-g2l9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:12:05.521760 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.521738 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2l9q\" (UniqueName: \"kubernetes.io/projected/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-kube-api-access-g2l9q\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:05.521760 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.521759 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:05.521913 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.521769 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7n9\" (UniqueName: \"kubernetes.io/projected/27c588bf-3b23-4ec3-91f7-fc6436d22e63-kube-api-access-zr7n9\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:05.521913 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:05.521778 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c588bf-3b23-4ec3-91f7-fc6436d22e63-plugins-config-volume\") on node \"ip-10-0-142-222.ec2.internal\" DevicePath \"\"" Aug 04 19:12:06.081550 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.081516 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" Aug 04 19:12:06.081550 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.081524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr" event={"ID":"e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa","Type":"ContainerDied","Data":"5e3a89fb4052ca1a9e8a21b7509711617a6f3df7f7fceceb291dae6edb078c16"} Aug 04 19:12:06.082057 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.081593 2575 scope.go:117] "RemoveContainer" containerID="c1ce60f05477afb9def6f2408fd1e207bba182a6db090f0d758a38bd043c2d5f" Aug 04 19:12:06.083110 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.083076 2575 generic.go:356] "Generic (PLEG): container finished" podID="27c588bf-3b23-4ec3-91f7-fc6436d22e63" containerID="4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5" exitCode=0 Aug 04 19:12:06.083205 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.083120 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" Aug 04 19:12:06.083205 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.083179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" event={"ID":"27c588bf-3b23-4ec3-91f7-fc6436d22e63","Type":"ContainerDied","Data":"4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5"} Aug 04 19:12:06.083309 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.083234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s" event={"ID":"27c588bf-3b23-4ec3-91f7-fc6436d22e63","Type":"ContainerDied","Data":"42f419833aaa0be49c01faeee0f7345381265be71bd35e82c1e89a343565d363"} Aug 04 19:12:06.084834 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.084800 2575 generic.go:356] "Generic (PLEG): container finished" podID="9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" containerID="ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250" exitCode=0 Aug 04 19:12:06.084946 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.084874 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" Aug 04 19:12:06.084946 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.084884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" event={"ID":"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7","Type":"ContainerDied","Data":"ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250"} Aug 04 19:12:06.084946 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.084923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b" event={"ID":"9e5e3c88-97fa-4c4a-a426-3b3d68e37df7","Type":"ContainerDied","Data":"b7374042515a1f4421fe54f0561ea49cd12ac18254c9b34053aa698d06f78eab"} Aug 04 19:12:06.091040 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.091022 2575 scope.go:117] "RemoveContainer" containerID="4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5" Aug 04 19:12:06.099556 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.099542 2575 scope.go:117] "RemoveContainer" containerID="4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5" Aug 04 19:12:06.099839 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:12:06.099820 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5\": container with ID starting with 4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5 not found: ID does not exist" containerID="4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5" Aug 04 19:12:06.099879 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.099845 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5"} err="failed to get container status \"4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5\": rpc error: code = NotFound desc = could not find container \"4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5\": container with ID starting with 4631509dfcfc65c3359dae0be5c47152340cc60b76e23d33961cc29b736245d5 not found: ID does not exist" Aug 04 19:12:06.099879 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.099862 2575 scope.go:117] "RemoveContainer" containerID="ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250" Aug 04 19:12:06.107697 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.107681 2575 scope.go:117] "RemoveContainer" containerID="ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250" Aug 04 19:12:06.107940 ip-10-0-142-222 kubenswrapper[2575]: E0804 19:12:06.107920 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250\": container with ID starting with ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250 not found: ID does not exist" containerID="ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250" Aug 04 19:12:06.107994 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.107948 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250"} err="failed to get container status \"ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250\": rpc error: code = NotFound desc = could not find container \"ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250\": container with ID starting with ca0de085397ccd35588801d4753ad27588ce395eb90d8ba53dac514fe363b250 not found: ID does not exist" Aug 04 19:12:06.122702 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.122681 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr"] Aug 04 19:12:06.137064 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.137014 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de2394-b45df858f-dqqkr"] Aug 04 19:12:06.155409 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.155387 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s"] Aug 04 19:12:06.162591 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.162567 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-de2394-65c9795d9d-bn95s"] Aug 04 19:12:06.191306 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.191278 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b"] Aug 04 19:12:06.194243 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:06.194223 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-de2394-58857c9d96-7gp8b"] Aug 04 19:12:07.349255 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:07.349222 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c588bf-3b23-4ec3-91f7-fc6436d22e63" path="/var/lib/kubelet/pods/27c588bf-3b23-4ec3-91f7-fc6436d22e63/volumes" Aug 04 19:12:07.349675 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:07.349658 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" path="/var/lib/kubelet/pods/9e5e3c88-97fa-4c4a-a426-3b3d68e37df7/volumes" Aug 04 19:12:07.350014 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:07.350001 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" path="/var/lib/kubelet/pods/e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa/volumes" Aug 04 19:12:19.992375 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:19.992333 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-ls9rb_e7690ee0-8a42-4266-8825-67a07d42b787/copy-manifests/0.log" Aug 04 19:12:20.012110 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:20.012082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-ls9rb_e7690ee0-8a42-4266-8825-67a07d42b787/manager/0.log" Aug 04 19:12:22.659903 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:22.659868 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-r6m99_27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9/manager/0.log" Aug 04 19:12:23.417512 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:23.417484 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b9fc5_76d16691-b496-43ea-960a-afac7fecc344/discovery/0.log" Aug 04 19:12:24.201497 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:24.201469 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fbf496bb-v6nzt_40806ebc-9194-4588-93ba-7f9563faaf25/router/0.log" Aug 04 19:12:24.555196 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:24.555122 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kq9zq_2dab5fb1-d3fb-44cb-8cd9-d0bbeffc81e8/main/0.log" Aug 04 19:12:24.932416 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:24.932356 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fz7fww_980f0560-6865-4966-8356-225ec2a7fee9/main/0.log" Aug 04 19:12:34.932080 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:34.932048 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4fkfc_c2d36207-38b9-4ace-a64d-f0439dea7bdc/global-pull-secret-syncer/0.log" Aug 04 19:12:35.180468 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:35.180439 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h6dbd_b51c8b78-a028-4ace-8861-369f61795c3b/konnectivity-agent/0.log" Aug 04 19:12:35.274283 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:35.274211 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-222.ec2.internal_90563c7e7eb521e876399cd564ee59a0/haproxy/0.log" Aug 04 19:12:40.602527 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:40.602489 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-r6m99_27b3f9a1-7f75-463c-bf41-9cdf6d0b77b9/manager/0.log" Aug 04 19:12:42.519437 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:42.519410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-h5r49_2363937d-6717-4808-85f9-de0fe414de74/cluster-monitoring-operator/0.log" Aug 04 19:12:42.593546 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:42.593519 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-866wn_b7384e93-e3d1-4ab3-bf64-1e932beac4f9/kube-state-metrics/0.log" Aug 04 19:12:42.631775 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:42.631749 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-866wn_b7384e93-e3d1-4ab3-bf64-1e932beac4f9/kube-rbac-proxy-main/0.log" Aug 04 19:12:42.686026 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:42.685998 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-866wn_b7384e93-e3d1-4ab3-bf64-1e932beac4f9/kube-rbac-proxy-self/0.log" Aug 04 19:12:42.730931 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:42.730895 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dbc4959c4-qh68s_360f78b4-3925-4b2e-ac2d-2c948347762e/metrics-server/0.log" Aug 04 19:12:42.965683 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:42.965605 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rf6pt_a8048dc7-8311-402c-a23e-1fcb4f629df8/node-exporter/0.log" Aug 04 19:12:43.007434 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:43.007406 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rf6pt_a8048dc7-8311-402c-a23e-1fcb4f629df8/kube-rbac-proxy/0.log" Aug 04 19:12:43.039294 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:43.039270 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rf6pt_a8048dc7-8311-402c-a23e-1fcb4f629df8/init-textfile/0.log" Aug 04 19:12:43.194948 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:43.194912 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-qhqvf_2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5/kube-rbac-proxy-main/0.log" Aug 04 19:12:43.232265 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:43.232194 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-qhqvf_2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5/kube-rbac-proxy-self/0.log" Aug 04 19:12:43.264794 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:43.264765 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-qhqvf_2e4b2eaf-5a17-456a-a8e6-8cec67a6e1b5/openshift-state-metrics/0.log" Aug 04 19:12:45.230027 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.229991 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p"] Aug 04 19:12:45.230417 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230355 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1723f4-0e6a-4d66-8ee6-8934fd518a02" containerName="payload-processing" Aug 04 19:12:45.230417 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230383 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1723f4-0e6a-4d66-8ee6-8934fd518a02" containerName="payload-processing" Aug 04 19:12:45.230417 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230398 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4272d631-b4f6-4c4d-968c-d68f95da1407" containerName="maas-api" Aug 04 19:12:45.230417 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230406 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4272d631-b4f6-4c4d-968c-d68f95da1407" containerName="maas-api" Aug 04 19:12:45.230417 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230413 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386b30fa-b42e-455d-97d1-a0665c1eb17a" containerName="payload-processing" Aug 04 19:12:45.230417 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230418 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="386b30fa-b42e-455d-97d1-a0665c1eb17a" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230426 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105a19a6-4014-48ec-8343-b705bc7b5dbe" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230431 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="105a19a6-4014-48ec-8343-b705bc7b5dbe" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230438 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" containerName="maas-api" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230443 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" containerName="maas-api" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230452 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34710ed6-65be-4d7c-8b50-2c4d3d581f83" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230457 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="34710ed6-65be-4d7c-8b50-2c4d3d581f83" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c23bff92-8351-4d4c-9cec-0a9045d5106f" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230469 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c23bff92-8351-4d4c-9cec-0a9045d5106f" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230476 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c588bf-3b23-4ec3-91f7-fc6436d22e63" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230481 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c588bf-3b23-4ec3-91f7-fc6436d22e63" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230490 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230496 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230504 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230515 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230520 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230526 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ff10c0-2667-47c7-abef-09728d4ece78" containerName="maas-api" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230530 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ff10c0-2667-47c7-abef-09728d4ece78" containerName="maas-api" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230594 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5e3c88-97fa-4c4a-a426-3b3d68e37df7" containerName="payload-pre-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230607 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1723f4-0e6a-4d66-8ee6-8934fd518a02" containerName="payload-processing" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230613 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4272d631-b4f6-4c4d-968c-d68f95da1407" containerName="maas-api" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230620 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ff10c0-2667-47c7-abef-09728d4ece78" containerName="maas-api" Aug 04 19:12:45.230614 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230625 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d91e22-7d94-42c3-8b41-e74e0dd8b6aa" containerName="maas-api" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230631 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="34710ed6-65be-4d7c-8b50-2c4d3d581f83" containerName="payload-pre-processing" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230637 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c23bff92-8351-4d4c-9cec-0a9045d5106f" containerName="payload-processing" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230643 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="90dcc1a0-f8fd-48ba-96ea-3da3b2a2b95b" containerName="payload-pre-processing" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230648 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="105a19a6-4014-48ec-8343-b705bc7b5dbe" containerName="payload-processing" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230654 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="386b30fa-b42e-455d-97d1-a0665c1eb17a" containerName="payload-processing" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230659 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c588bf-3b23-4ec3-91f7-fc6436d22e63" containerName="payload-processing" Aug 04 19:12:45.231259 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.230664 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a723d6e-8dc1-42ee-a7f1-22e9e7cda459" containerName="payload-pre-processing" Aug 04 19:12:45.235122 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.235104 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.237661 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.237624 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kwzfh\"/\"openshift-service-ca.crt\"" Aug 04 19:12:45.237784 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.237734 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kwzfh\"/\"default-dockercfg-gz4dk\"" Aug 04 19:12:45.238566 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.238548 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kwzfh\"/\"kube-root-ca.crt\"" Aug 04 19:12:45.246783 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.246735 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p"] Aug 04 19:12:45.365021 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.364987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-lib-modules\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.365185 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.365030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-sys\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.365185 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.365115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-proc\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.365185 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.365148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxwp\" (UniqueName: \"kubernetes.io/projected/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-kube-api-access-slxwp\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.365293 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.365209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-podres\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465693 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-podres\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-lib-modules\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-sys\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-proc\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxwp\" (UniqueName: \"kubernetes.io/projected/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-kube-api-access-slxwp\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-lib-modules\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-proc\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.465886 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-sys\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.466138 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.465824 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-podres\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.476823 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.476791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxwp\" (UniqueName: \"kubernetes.io/projected/b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22-kube-api-access-slxwp\") pod \"perf-node-gather-daemonset-p9t2p\" (UID: \"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22\") " pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.545846 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.545767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:45.684178 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:45.684150 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p"] Aug 04 19:12:45.686502 ip-10-0-142-222 kubenswrapper[2575]: W0804 19:12:45.686477 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb8b9e8bf_1a2d_42fd_8ce9_ba3d281fbf22.slice/crio-ca17b06a7dca2a7acb81f48d2c423eaf090128f93859c193dd2da79c20839917 WatchSource:0}: Error finding container ca17b06a7dca2a7acb81f48d2c423eaf090128f93859c193dd2da79c20839917: Status 404 returned error can't find the container with id ca17b06a7dca2a7acb81f48d2c423eaf090128f93859c193dd2da79c20839917 Aug 04 19:12:46.094580 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.094541 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/1.log" Aug 04 19:12:46.102140 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.102112 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-8hh79_a154ea0c-86e0-4615-8314-450a19397b28/console-operator/2.log" Aug 04 19:12:46.300915 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.300875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" event={"ID":"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22","Type":"ContainerStarted","Data":"2aa511543a14eaa78364b72614903a26712620a8d8059299f49f63dce114821b"} Aug 04 19:12:46.301429 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.300927 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" event={"ID":"b8b9e8bf-1a2d-42fd-8ce9-ba3d281fbf22","Type":"ContainerStarted","Data":"ca17b06a7dca2a7acb81f48d2c423eaf090128f93859c193dd2da79c20839917"} Aug 04 19:12:46.301429 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.300963 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:46.323760 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.323710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" podStartSLOduration=1.3236947749999999 podStartE2EDuration="1.323694775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:12:46.321042556 +0000 UTC m=+3731.467858827" watchObservedRunningTime="2026-08-04 19:12:46.323694775 +0000 UTC m=+3731.470511034" Aug 04 19:12:46.746129 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:46.746097 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-dfb67b584-mh89d_b76101e4-4f7b-457f-8877-607fde9266e9/console/0.log" Aug 04 19:12:47.494002 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:47.493975 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-s7tzm_1fb0ddae-2f89-4c20-b7dc-12822928ddb6/volume-data-source-validator/0.log" Aug 04 19:12:48.661870 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:48.661839 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cvcw2_b3935d47-1a3e-4d52-b644-62a4fe867dfb/dns/0.log" Aug 04 19:12:48.689708 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:48.689684 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cvcw2_b3935d47-1a3e-4d52-b644-62a4fe867dfb/kube-rbac-proxy/0.log" Aug 04 19:12:48.896280 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:48.896250 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjjg9_5f8c2fcb-e227-4861-8e7a-5152a4ed3d24/dns-node-resolver/0.log" Aug 04 19:12:49.575081 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:49.575041 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xn6wp_b4763dc2-4f1b-4be6-a14b-0bdb04fd9206/node-ca/0.log" Aug 04 19:12:50.901682 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:50.901647 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b9fc5_76d16691-b496-43ea-960a-afac7fecc344/discovery/0.log" Aug 04 19:12:51.183224 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:51.183195 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fbf496bb-v6nzt_40806ebc-9194-4588-93ba-7f9563faaf25/router/0.log" Aug 04 19:12:52.035721 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.035681 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-94wjp_b55763a8-3b0c-4f5a-8e10-bc6964591a81/serve-healthcheck-canary/0.log" Aug 04 19:12:52.317255 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.317179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kwzfh/perf-node-gather-daemonset-p9t2p" Aug 04 19:12:52.650561 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.650465 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-m7zkx_b36af2a3-d7d0-469a-a78d-c20e9ffcb746/insights-operator/0.log" Aug 04 19:12:52.652076 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.652060 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-m7zkx_b36af2a3-d7d0-469a-a78d-c20e9ffcb746/insights-operator/1.log" Aug 04 19:12:52.763590 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.763560 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k5c8f_e45e02c6-0783-4ecd-ae77-1ada65667e4c/kube-rbac-proxy/0.log" Aug 04 19:12:52.790947 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.790918 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k5c8f_e45e02c6-0783-4ecd-ae77-1ada65667e4c/exporter/0.log" Aug 04 19:12:52.819320 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:52.819292 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k5c8f_e45e02c6-0783-4ecd-ae77-1ada65667e4c/extractor/0.log" Aug 04 19:12:55.732979 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:55.732950 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-ls9rb_e7690ee0-8a42-4266-8825-67a07d42b787/manager/0.log" Aug 04 19:12:55.812935 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:55.812907 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76bbc69bc6-ls9rb_e7690ee0-8a42-4266-8825-67a07d42b787/copy-manifests/0.log" Aug 04 19:12:56.404172 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:56.404136 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b9fc5_76d16691-b496-43ea-960a-afac7fecc344/discovery/0.log" Aug 04 19:12:56.551405 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:56.551274 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fbf496bb-v6nzt_40806ebc-9194-4588-93ba-7f9563faaf25/router/0.log" Aug 04 19:12:57.479703 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:57.479674 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84559d8974-dn8bf_261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e/manager/1.log" Aug 04 19:12:57.480228 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:57.480084 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84559d8974-dn8bf_261dbaaf-da0a-4ba7-8ab0-dfd5bc07733e/manager/0.log" Aug 04 19:12:58.355778 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:58.355697 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b9fc5_76d16691-b496-43ea-960a-afac7fecc344/discovery/0.log" Aug 04 19:12:58.489672 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:12:58.489639 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fbf496bb-v6nzt_40806ebc-9194-4588-93ba-7f9563faaf25/router/0.log" Aug 04 19:13:03.159620 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:13:03.159584 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v666v_f8d6a004-71d5-46dc-b7bb-06bfa86f8131/kube-storage-version-migrator-operator/0.log" Aug 04 19:13:03.168992 ip-10-0-142-222 kubenswrapper[2575]: I0804 19:13:03.168966 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-v666v_f8d6a004-71d5-46dc-b7bb-06bfa86f8131/kube-storage-version-migrator-operator/1.log"